{"ID":96683,"post_author":"9412100","post_date":"2021-04-29 15:40:49","post_date_gmt":"0000-00-00 00:00:00","post_content":"","post_title":"LIMSjournal - Spring 2021","post_excerpt":"","post_status":"draft","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"","to_ping":"","pinged":"","post_modified":"2021-04-29 15:40:49","post_modified_gmt":"2021-04-29 19:40:49","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.limsforum.com\/?post_type=ebook&p=96683","menu_order":0,"post_type":"ebook","post_mime_type":"","comment_count":"0","filter":"","_ebook_metadata":{"enabled":"on","private":"0","guid":"766BF62D-0C62-4CB6-B9AB-37C5BB29F7B1","title":"LIMSjournal - Spring 2021","subtitle":"Volume 7, Issue 1","cover_theme":"nico_7","cover_image":"https:\/\/www.limsforum.com\/wp-content\/plugins\/rdp-ebook-builder\/pl\/cover.php?cover_style=nico_7&subtitle=Volume+7%2C+Issue+1&editor=Shawn+Douglas&title=LIMSjournal+-+Spring+2021&title_image=https%3A%2F%2Fs3.limsforum.com%2Fwww.limsforum.com%2Fwp-content%2Fuploads%2FFig8_Liscouski_LabTechPlanMan20.png&publisher=LabLynx+Press","editor":"Shawn Douglas","publisher":"LabLynx Press","author_id":"26","image_url":"","items":{"e0147011cc1eb892e1a35e821657a6d9_type":"article","e0147011cc1eb892e1a35e821657a6d9_title":"Considerations in the automation of laboratory procedures (Liscouski 2021)","e0147011cc1eb892e1a35e821657a6d9_url":"https:\/\/www.limswiki.org\/index.php\/LII:Considerations_in_the_Automation_of_Laboratory_Procedures","e0147011cc1eb892e1a35e821657a6d9_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tLII:Considerations in the Automation of Laboratory Procedures\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tTitle: Considerations in the Automation of Laboratory Procedures\nAuthor for citation: Joe Liscouski, with editorial modifications by Shawn Douglas\nLicense for content: Creative Commons Attribution 4.0 International\nPublication date: January 2021\n\nContents\n\n1 Introduction \n2 How does this discussion relate to previous work? \n\n2.1 Before we get too far into this... \n\n\n3 Transitioning from typical lab operations to automated systems \n\n3.1 What will happen to people\u2019s jobs as a result of automation? \n3.2 What is the role of AI and ML in automation? \n3.3 Where do we find the resources to carry out automation projects\/programs? \n3.4 What equipment would we need for automated processes, and will it be different that what we currently have? \n3.5 What role does a LES play in laboratory automation? \n\n3.5.1 What does this have to do with automation? \n\n\n3.6 How do we go about planning for automation? \n\n3.6.1 Justification, expectations, and goals \n3.6.2 Analyzing the process \n3.6.3 Scheduling automation projects \n3.6.4 Budgeting \n\n\n\n\n4 Build, buy, or cooperate? \n5 Project planning \n6 Conclusions (so far) \n7 Abbreviations, acronyms, and initialisms \n8 Footnotes \n9 About the author \n10 References \n\n\n\nIntroduction \nScientists have been dealing with the issue of laboratory automation for decades, and during that time the meaning of those words has expanded from the basics of connecting an instrument to a computer, to the possibility of a fully integrated informatics infrastructure beginning with sample preparation and continuing on to the laboratory information management system (LIMS), electronic laboratory notebook (ELN), and beyond. Throughout this evolution there has been one underlying concern: how do we go about doing this?\nThe answer to that question has changed from a focus on hardware and programming, to today\u2019s need for a lab-wide informatics strategy. We\u2019ve moved from the bits and bytes of assembly language programming to managing terabytes of files and data structures.\nThe high-end of the problem\u2014the large informatics database systems\u2014has received significant industry-wide attention in the last decade. The stuff on the lab bench, while the target of a lot of individual products, has been less organized and more experimental. Failed or incompletely met promises have to yield to planned successes. How we do it needs to change. This document is about the considerations required when making that change. The haphazard \"let's try this\" method has to give way to more engineered solutions and a realistic appraisal of the human issues, as well as the underlying technology management and planning.\nWhy is this important? Whether you are conducting intense laboratory experiments to produce data and information or making chocolate chip cookies in the kitchen, two things remain important: productivity and the quality of the products. In either case, if the productivity isn\u2019t high enough, you won\u2019t be able to justify your work; if the quality isn\u2019t there, no one will want what you produce. Conducting laboratory work and making cookies have a lot in common. Your laboratories exist to answer questions. What happens if I do this? What is the purity of this material? What is the structure of this compound? The field of laboratories asking these questions is extensive, basically covering the entire array of lab bench and scientific work, including chemistry, life sciences, physics, and electronics labs. The more efficiently we answer those questions, the more likely it will be that these labs will continue operating and, that you\u2019ll achieve the goals your organization has set. At some point, it comes down to performance against goals and the return on the investment organizations make in lab operations.\nIn addition to product quality and productivity, there are a number of other points that favor automation over manual implementations of lab processes. They include:\n\n lower costs per test;\n better control over expenditures;\n a stronger basis for better workflow planning;\n reproducibility;\n predictably; and\n tighter adherence to procedures, i.e., consistency.\nLists similar to the one above can be found in justifications for lab automation, and cookie production, without further comment. It\u2019s just assumed that everyone agrees and that the reasoning is obvious. Since we are going to use those items to justify the cost and effort that goes into automation, we should take a closer look at them.\nLets begin with reproducibility, predictability, and consistency, very similar concerns that reflect automation\u2019s ability to produce the same product with the desired characteristics over and over. For data and information, that means that the same analysis on the same materials will yield the same results, that all the steps are documented and that the process is under control. The variability that creeps into the execution of a process by people is eliminated. That variability in human labor can result from the quality of training, equipment setup and calibration, readings from analog devices (e.g., meters, pipette meniscus, charts, etc.), there is a long list of potential issues.\nConcerns with reproducibility, predictability, and consistency are common to production environments, general lab work, manufacturing, and even food service. There are several pizza restaurants in our area using one of two methods of making the pies. Both start the preparation the same way, spreading dough and adding cheese and toppings, but the differences are in how they are cooked. Once method uses standard ovens (e.g., gas, wood, or electric heating); the pizza goes in, the cook watches it, and then removes it when the cooking is completed. This leads to a lot of variability in the product, some a function of the cook\u2019s attention, some depending on requests for over or under cooking the crust. Some is based on \"have it your way\" customization. The second method uses a metal conveyor belt to move the pie through an oven. The oven temperature is set as is the speed of the belt, and as long as the settings are the same, you get a reproducible, consistent product order after order. It\u2019s a matter of priorities. Manual verses automated. Consistent product quality verses how the cook feels that day. In the end, reducing variability and being able to demonstrate consistent, accurate, results gives people confidence in your product.\nLower costs per test, better control over expenditures, and better workflow planning also benefit from automation. Automated processes are more cost-efficient since the sample throughput is higher and the labor cost is reduced. The cost per test and the material usage is predictable since variability in components used in testing is reduced or eliminated, and workflow planning is improved since the time per test is known, work can be better scheduled. Additionally, process scale-up should be easier if there is a high demand for particular procedures. However there is a lot of work that has to be considered before automation is realizable, and that is where this discussion is headed.\n\nHow does this discussion relate to previous work? \nThis work follows on the heels of two previous works:\n\n Computerized Systems in the Modern Laboratory: A Practical Guide (2015): This book presents the range of informatics technologies, their relationship to each other, and the role they play in laboratory work. It differentiates a LIMS from an ELN and scientific data management system (SDMS) for example, contrasting their use and how they would function in different lab working environments. In addition, it covers topics such as support and regulatory issues.\n A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work (2018): This webinar series complements the above text. It begins by introducing the major topics in informatics (e.g., LIMS, ELN, etc.) and then discusses their use from a strategic viewpoint. Where and how do you start planning? What is your return on investment? What should get implemented first, and then what are my options? The series then moves on to developing an information management strategy for the lab, taking into account budgets, support, ease of implementation, and the nature of your lab\u2019s work.\nThe material in this write-up picks up where the last part of the webinar series ends. The last session covers lab processes, amd this picks up that thread and goes into more depth concerning a basic issue: how do you move from manual methods to automated systems?\nProductivity has always been an issue in laboratory work. Until the 1950s, a lab had little choice but to add more people if more work needed to be done. Since then, new technologies have afforded wider options, including new instrument technologies. The execution of the work was still done by people, but the tools were better. Now we have other options. We just have to figure out when, if, and how to use them.\n\nBefore we get too far into this... \nWith elements such as productivity, return on investment (ROI), data quality, and data integrity as driving factors in this work, you shouldn\u2019t be surprised if a lot of the material reads like a discussion of manufacturing methodologies; we\u2019ve already seen some examples. We are talking about scientific work, but the same things that drive the elements noted in labs have very close parallels in product manufacturing. The work we are describing here will be referenced as \"scientific manufacturing,\" manufacturing or production in support of scientific programs.[a]\nThe key points of a productivity conversation in both lab and material production environments are almost exact overlays, the only significant difference is that the results of the efforts are data and information in one case, and a physical item you might sell in the other. Product quality and integrity are valued considerations in both. For scientists, this may require an adjustment to their perspectives when dealing with automation. On the plus side, the lessons learned in product manufacturing can be applied to lab bench work, making the path to implementation a bit easier while providing a framework for understanding what a successful automation effort looks like. People with backgrounds in product manufacturing can be a useful resource in the lab, with a bit of an adjustment in perspective on their part.\n\nTransitioning from typical lab operations to automated systems \nTransitioning a lab from its current state of operations to one that incorporates automation can raise a number of questions, and people\u2019s anxiety levels. There are several questions that should be considered to set expectations for automated systems and how they will impact jobs and the introduction of new technologies. They include:\n\n What will happen to people\u2019s jobs as a result of automation?\n What is the role of artificial intelligence (AI) and machine learning (ML) in automation?\n Where do we find the resources to carry out automation projects\/programs?\n What equipment would we need for automated processes, and will it be different that what we currently have?\n What role does a laboratory execution system (LES) play in laboratory automation?\n How do we go about planning for automation?\nWhat will happen to people\u2019s jobs as a result of automation? \nStories are appearing in print, online, and in television news reporting about the potential for automation to replace human effort in the labor force. It seems like it is an all-or-none situation, either people will continue working in their occupations or automation (e.g., mechanical, software, AI, etc.) will replace them. The storyline is people are expensive and automated work can be less costly in the long run. If commercial manufacturing is a guide, automation is a preferred option from both a productivity and an ROI perspective. In order to make the productivity gains from automation similar to those seen in commercial manufacturing, there are some basic requirements and conditions that have to be met:\n\n The process has to be well documented and understood, down to the execution of each step without variation, while error detection and recovery have to be designed in.\n The process has to remain static and be expected to continue over enough execution cycles to make it economically attractive to design, build, and maintain.\n Automation-compatible equipment has to be available. Custom-built components are going to be expensive and could represent a barrier to successful implementation.\n There has to be a driving need to justify the cost of automation; economics, the volume of work that has to be addressed, working with hazardous materials, and lack of educated workers are just a few of the factors that would need to be considered.\nThere are places in laboratory work where production-scale automation has been successfully implemented; life sciences applications for processes based on microplate technologies are one example. When we look at the broad scope of lab work across disciplines, most lab processes don\u2019t lend themselves to that level of automation, at least not yet. We\u2019ll get into this in more detail later. But that brings us back to the starting point: what happens to people's jobs?\nIn the early stages of manufacturing automation, as well as fields such as mining where work was labor intensive and repetitive, people did lose jobs when new methods of production were introduced. That shift from a human workforce to automated task execution is expanding as system designers probe markets from retail to transportation.[1] Lower skilled occupations gave way first, and we find ourselves facing automation efforts that are moving up the skills ladder, most recently is the potential for automated driving, a technology that has yet to be fully embraced but is moving in that direction. The problem that leaves us with is providing displaced workers with a means of employment that gives them at least a living income, and the purpose, dignity, and self-worth that they\u2019d like to have. This is going to require significant education, and people are going to have to come to grips with the realization that education never stops.\nDue to the push for increased productivity, lab work has seen some similar developments in automation. The development of automated pipettes, titration stations, auto-injectors, computer-assisted instrumentation, and automation built to support microplate technologies represent just a few places where specific tasks have been addressed. However these developments haven\u2019t moved people out of the workplace as has happened in manufacturing, mining, etc. In some cases they\u2019ve changed the work, replacing repetitive time-consuming tasks with equipment that allows lab personnel to take on different tasks. In other cases the technology addresses work that couldn\u2019t be performed in a cost-effective manner with human effort; without automation, that work might just not be feasible due to the volume of work (whose delivery might be limited by the availability of the right people, equipment, and facilities) or the need to work with hazardous materials. Automation may prevent the need for hiring new people while giving those currently working more challenging tasks.\nAs noted in the previous paragraph, much of the automation in lab work is at the task level: equipment designed to carry out a specific function such as Karl-Fisher titrations. Some equipment designed around microplate formats can function at both the task level and as part of user-integrated robotics system. This gives the planner useful options about the introduction of automation that makes it easier for personnel to get accustomed to automation before moving into scientific manufacturing.\nOverall, laboratory people shouldn\u2019t be loosing their jobs as a result of lab automation, but they do have to be open to changes in their jobs, and that could require an investment in their education. Take someone whose current job is to carry out a lab procedure, someone who understands all aspects of the work, including troubleshooting equipment, reagents, and any special problems that may crop up. Someone else may have developed the procedure, but that person is the expert in its execution.\nFirst of all you need these experts to help plan and test the automated systems if you decide to create that project. These would also be the best people to educate as automated systems managers; they know how the process is supposed to work and should be in a position to detect problems. If it crashes, you\u2019ll need someone who can cover the work while problems are be addressed. Secondly, if lab personnel get the idea that they are watching their replacement being installed, they may leave before the automated systems are ready. In the event of a delay, you\u2019ll have a backlog and no one to handle it.\nBeyond that, people will be freed from the routine of carrying out processes and be able to address work that had been put on a back burner until it could be addressed. As we move toward automated systems, jobs will change by expansion to accommodate typical lab work, as well as the management, planning, maintenance, and evolution of laboratory automation and computing.\nAutomation in lab work is not an \"all or none\" situation. Processes can be structured so that the routine work is done by systems, and the analyst can spend time reviewing the results, looking for anomalies and interesting patterns, while being able to make decisions about the need for and nature of follow-on efforts.\n\nWhat is the role of AI and ML in automation? \nWhen we discuss automation, what we are referencing now is basic robotics and programming. AI may, and likely will, play a role in the work, but first we have to get the foundations right before we consider the next step; we need to put in the human intelligence first. Part of the issue with AI is that we don\u2019t know what it is.\nScience fiction aside, many of today's applications of AI have a limited role in lab work today. Here are some examples:\n\n Having a system that can bring up all relevant information on a research question\u2014a sort of super Google\u2014or a variation of IBM\u2019s Watson could have significant benefits.\n Analyzing complex data or large volumes of data could be beneficial, e.g., the analysis of radio astronomy data to find fast radio bursts (FRB). After discovering 21 FRB signals upon analyzing five hours of data, researchers at Green Bank Telescope used AI to analyze 400 terabytes of older data and detected another 100.[2]\n \"[A] team at Glasgow University has paired a machine-learning system with a robot that can run and analyze its own chemical reaction. The result is a system that can figure out every reaction that's possible from a given set of starting materials.\"[3]\n HelixAI is using Amazon's Alexa as a digital assitant for laboratory work.[4]\nNote that the points above are research-based applications, not routine production environments where regulatory issues are important. While there are research applications that might be more forgiving of AI systems because the results are evaluated by human intelligence, and problematic results can be made subject to further verification, data entry systems such as voice entry have to be carefully tested and the results of that data entry verified and shown to be correct.\nPharma IQ continues to publish material on advanced topics in laboratory informatics, including articles on how labs are benefiting from new technologies[5] and survey reports such as AI 2020: The Future of Drug Discovery. In that report they note[6]:\n\n \"94% of pharma professionals expect that intelligent technologies will have a noticeable impact on the pharmaceutical industry over the next two years.\"\n \"Almost one fifth of pharma professionals believe that we are on the cusp of a revolution.\"\n \"Intelligent automation and predictive analytics are expected to have the most significant impact on the industry.\"\n \"However, a lack of understanding and awareness about the benefits of AI-led technologies remain a hindrance to their implementation.\"\nNote that these are expectations, not a reflection of current reality. That same report makes comments about the impact of AI on headcount disruption, asking, \"Do you expect intelligent enterprise technologies[b] to significantly cut and\/or create jobs in pharma through 2020?\" Among the responses, 47 percent said they expected those technologies to do both, 40 percent said it will create new job opportunities, and 13 percent said there will be no dramatic change, with zero percent saying they expected solely job losses.[6]\nWhile there are high levels of expectations and hopes for results, we need to approach the idea of AI in labs with some caution. We read about examples based on machine learning (ML), for example using computer systems to recognize cats in photos, to recognized faces in a crowd, etc. We don\u2019t know how they accomplish their tasks, and we can\u2019t analyze their algorithms and decision-making. That leaves us with testing in quality, which at best is an uncertain process with qualified results (it has worked so far). One problem with testing AI systems based on ML is that they are going to continually evolve, so testing may affect the ML processes by providing a bias. It may also cause continued, redundant testing, because something we thought was evaluated was changed by the \u201cexperiences\u201d the AI based it\u2019s learning on. As one example, could the AI modify the science through process changes without our knowing because it didn\u2019t understand the science or the goals of the work?\nAI is a black box with ever-changing contents. That shouldn\u2019t be taken as a condemnation of AI in the lab, but rather as a challenge to human intelligence in evaluating, proving, and applying the technology. That application includes defining the operating boundaries of an AI system. Rather than creating a master AI for a complete process, we may elect to divide the AI\u2019s area of operation into multiple, independent segments, with segment integration occurring in later stages once we are confident in their ability to work and show clear evidence of systems stability. In all of this we need to remember that our goal is the production of high-quality data and information in a controlled, predictable environment, not gee-wiz technology. One place where AI (or clever programming) could be of use is in better workflow planning, which takes into account current workloads and assignments, factors in the inevitable panic-level testing need, and, perhaps in a QC\/production environment, anticipates changes in analysis requirements based on changes in production operations.\nThroughout this section I've treated \u201cAI\u201d as \u201cartificial intelligence,\u201d its common meaning. There may be a better way of looking at it for lab use as, noted in this excerpt from the October 2018 issue of Wired magazine[7]:\n\nAugmented intelligence. Not \u201cartificial,\u201d but how Doug Engelbart[c] envisioned our relationship with computer: AI doesn\u2019t replace humans. It offers idiot-savant assistants that enable us to become the best humans we can be.\nAugmented intelligence (AuI) is a better term for what we might experience in lab work, at least in the near future. It suggests something that is both more realistic and attainable, with the synergism that would make it, and automation, attractive to lab management and personnel\u2014a tool they can work with and improve lab operations that doesn\u2019t carry the specter of something going on that they don\u2019t understand or control. OPUS\/SEARCH from Bruker might be just such an entry in this category.[8] AuI may serve as a first-pass filter for large data sets\u2014as noted in the radio astronomy and chemistry examples noted earlier\u2014reducing those sets of data and information to smaller collections that human intelligence can\/should evaluate. However, that does put a burden on the AuI to avoid excessive false positives or negatives, something that can be adjusted over time.\nBeyond that there is the possibility of more cooperative work between people and AuI systems. An article in Scientific American titled \u201cMy Boss the Robot\u201d[9] describes the advantage of a human-robot team, with the robot doing the heavy work and the human\u2014under the robots guidance\u2014doing work he was more adept at, verses a team of experts with the same task. The task, welding a Humvee frame, was competed by the human machine pair in 10 hours at a cost of $1,150; the team of experts took 89 hours and a cost of $7,075. That might translate into terms of laboratory work by having a robot do routine, highly repetitive tasks and the analyst overseeing the operation and doing higher-level analysis of the results.\nCertainly, AI\/AuI is going to change over time as programming and software technology becomes more sophisticated and capable; today\u2019s example of AuI might be seen as tomorrow\u2019s clever software. However, a lot depends on the experience of the user.\nThere is something important to ask about laboratory technology development, and AI in particular: is the direction of development going to be the result of someone\u2019s innovation that people look at and embrace, or will it be the result of a deliberate choice of lab people saying \u201cthis is where we need to go, build systems that will get us there\u201d? The difference is important, and lab managers and personnel need to be in control of the planning and implementation of systems.\n\nWhere do we find the resources to carry out automation projects\/programs? \nGiven the potential scope of work, you may need people with skills in programming, robotics, instrumentation, and possibly mechanical or electrical engineering if off-the-shelf components aren\u2019t available. The biggest need is for people who can do the planning and optimization that is needed as you move from manual to semi- or fully-automated systems, particularly specialists in process engineering who can organize and plan the work, including the process controls and provision for statistical process control.\nWe need to develop people who are well versed in laboratory work and the technologies that can be applied to that work, as assets in laboratory automation development and planning. In the past, this role has been filled with lab personnel having an interest in the subject, IT people willing to extend their responsibilities, and\/or outside consultants. A 2017 report by Salesforce Research states \"77% of IT leaders believe IT functions as an extension\/partner of business units rather than as a separate function.\"[10] The report makes no mention of laboratory work or manufacturing aside from those being functions within businesses surveyed. Unless a particular effort is made, IT personnel rarely have the backgrounds needed to meet the needs of lab work. In many cases, they will try and fit lab needs into software they are already familiar with, rather then extend their backgrounds into new computational environments. Office and pure database applications are easily handled, but when we get to the lab bench, it's another matter entirely.\nThe field is getting complex enough that we need people whose responsibilities span both science and technology. This subject is discussed in the webinar series A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work, Part 5 \"Supporting Laboratory Systems.\"\n\nWhat equipment would we need for automated processes, and will it be different that what we currently have? \nThis is an interesting issue and it directly addresses the commitment labs have to automation, particularly robotics. In the early days of lab automation when Zymark (Zymate and Benchmate), Perkin Elmer, and Hewlett Packard (ORCA) were the major players in the market, the robot had to adapt to equipment that was designed for human use: standard laboratory equipment. They did that through special modifications and the use of different grippers to handle test tubes, beakers, and flasks. While some companies wanted to test the use of robotics in the lab, they didn\u2019t want to invest in equipment that could only be used with robots; they wanted lab workers to pick up where the robots left off in case the robots didn\u2019t work.\nSince then, equipment has evolved to support automation more directly. In some cases it is a device (e.g., a balance, pH meter, etc.) that has front panel human operator capability and rear connectors for computer communications. Liquid handling systems have seen the most advancement through the adoption of microplate formats and equipment designed to work with them. However, the key point is standardization of the sample containers. Vials and microplates lend themselves to a variety of automation devices, from sample processing to auto-injectors\/samplers. The issue is getting the samples into those formats.\nOne point that labs, in any scientific discipline, have to come to grips with is the commitment to automation. That commitment isn\u2019t going to be done on a lab-wide basis, but on a procedure-by-procedure basis. Full automation may not be appropriate for all lab work, whereas partial automation may be a better choice, and in some cases no automation may be required (we\u2019ll get into that later). The point that needs to be addressed is the choice of equipment. In most cases, equipment is designed for use by people, with options for automation and electronic communications. However, if you want to maximize throughput, you may have to follow examples from manufacturing and commit to equipment that is only used by automation. That will mean a redesign of the equipment, a shared risk for both the vendors and the users. The upside to this is that equipment can be specifically designed for a task, be more efficient, have the links needed for integration, use less material, and, more likely, take up less space. One example is the microplate, allowing for tens, hundreds, or thousands (depending on the plate used) of sample cells in a small space. What used to take many cubic feet of space as test tubes (the precursor to using microplates) is now a couple of cubic inches, using much less material and working space. Note, however, that while microplates are used by lab personnel, their use in automated systems provides greater efficiency and productivity.\nThe idea of equipment used only in an automated process isn\u2019t new. The development and commercialization of segmented flow analyzers\u2014initially by Technicon in the form of the AutoAnalyzers for general use, and the SMA (Sequential Multiple Analyzer) and SMAC (Sequential Multiple Analyzer with Computer) in clinical markets\u2014improved a lab's ability to process samples. These systems were phased out with new equipment that consumed less material. Products like these are being provided by Seal Analytical[11] for environmental work and Bran+Luebbe (a division of SPX Process Equipment in Germany).[12]\nThe issue in committing to automated equipment is that vendors and users will have to agree on equipment specifications and use them within procedures. One place this has been done successfully is in clinical chemistry labs. What other industry workflows could benefit? Do the vendors lead or do the users drive the issue? Vendors need to be convinced that there is a viable market for product before making an investment, and users need to be equally convinced that they will succeed in applying those products. In short, procedures that are important to a particular industry have to be identified, and both users and vendors have to come together to develop automated procedure and equipment specifications for products. This has been done successfully in clinical chemistry markets to the extent that equipment is marketed for use as validated for particular procedures.\n\nWhat role does a LES play in laboratory automation? \nBefore ELNs settled into their current role in laboratory work, the initial implementations differed considerably from what we have now. LabTech Notebook was released in 1986 (discontinued in 2004) to provide communications between computers and devices that used RS-232 serial communications. In the early 2000s SmartLab from Velquest was the first commercial product to carry the \"electronic laboratory notebook\" identifier. That product became a stand-alone entry in the laboratory execution system (LES) market; since its release, the same conceptual functionality has been incorporated into LIMS and ELNs that fit the more current expectation for an ELN.\nAt it\u2019s core, LES are scripted test procedures that an analyst would follow to carry out a laboratory method, essentially functioning as the programmed execution of a lab process. Each step in a process is described, followed exactly, and provision is made within the script for data collection. In addition, the LES can\/will (depending on the implementation; \"can\" in the case of SmartLab) check to see if the analyst is qualified to carry out the work and that the equipment and reagents are current, calibrated, and suitable for use. The systems can also have access to help files that an analyst can reference if there are questions about how to carry out a step or resolve issues. Beyond that, the software had the ability to work with lab instruments and automatically acquire data either through direct interfaces (e.g., balances, pH meters, etc.) or through parsing PDF files of instrument reports.\nThere are two reasons that these systems are attractive. First, they provide for a rigorous execution of a process with each step being logged as it is done. Second, that log provides a regulatory inspector with documented evidence that the work was done properly, making it easier for the lab to meet any regulatory burden.\nSince the initial development of SmartLab, that product has changed ownership and is currently in the hands of Dassault Syst\u00e8mes as part of the BIOVIA product line. As noted above, LIMS and ELN vendors have incorporated similar functionality into their products. Using those features requires \u201cscripting\u201d (in reality, software development), but it does allow the ability to access the database structures within those products. The SmartLab software needed programmed interfaces to other vendors' LIMS and ELNs to gain access to the same information.\n\nWhat does this have to do with automation? \nWhen we think about automated systems, particularly full-automation with robotic support, it is a programmed process from start to finish. The samples are introduced at the start, and the process continues until the final data\/information is reported and stored. These can be large scale systems using microplate formats, including tape-based systems from Douglas Scientific[13], programmable autosamplers such as those from Agilent[14], or systems built around robotics arms from a variety of vendors that move samples from one station to another.\nBoth LES and the automation noted in the previous paragraph have the following point in common: there is a strict process that must be followed, with no provision for variation. The difference is that in one case that process is implemented completely through the use of computers, as well as electronic and mechanical equipment. In the other case, the process is being carried out by lab personnel using computers, as well as electronic and mechanical lab equipment. In essence, people take the place of mechanical robots, which conjures up all kinds of images going back to the 1927 film Metropolis.[d] Though the LES represents a step toward more sophisticated automation, both methods still require:\n\n programming, including \u201cscripting\u201d (the LES methods are a script that has to be followed);\n validated, proven processes; and\n qualified staff, though the qualifications differ. (In both cases they have to be fully qualified to carry out the process in question. However in the full automation case, they will require more education on running, managing, and troubleshooting the systems.)\nIn the case of full automation, there has to be sufficient justification for the automation of the process, including sufficient sample processing. The LES-human implementation can be run for a single sample if needed, and the operating personnel can be trained on multiple procedures, switching tasks as needed. Electro-mechanical automation would require a change in programming, verification that the system is operating properly, and may require equipment re-configuration. Which method is better for a particular lab depends on trade-offs between sample load, throughput requirements, cost, and flexibility. People are adaptable, easily moving between tasks, whereas equipment has to be adapted to a task.\n\nHow do we go about planning for automation? \nThere are three forms of automation to be considered:\n\n No automation \u2013 Instead, the lab relies on lab personnel to carry out all steps of a procedure.\n Partial automation \u2013 Automated equipment is used to carry out steps in a procedure. Given the current state of laboratory systems, this is the most prevalent since most lab equipment has computer components in them to facilitate their use.\n Full automation - The entire process is automated. The definition of \u201centire\u201d is open to each labs interpretation and may vary from one process to another. For example, some samples may need some handing before they are suitable for use in a procedure. That might be a selection process from a freezer, grinding materials prior to a solvent extraction, and so on, representing cases where the equipment available isn\u2019t suitable for automated equipment interaction. One goal is to minimize this effort since it can put a limit on the productivity of the entire process. This is also an area where negotiation between the lab and the sample submitter can be useful. Take plastic pellets for example, which often need to be ground into a course powder before they can be analyzed; having the submitter provide them in this form will reduce the time and cost of the analysis. Standardizing on the sample container can also facilitate the analysis (having the lab provide the submitter with standard sample vials using barcodes or RFID chips can streamline the process).\nOne common point that these three forms share is a well-described method (procedure, process) that needs to be addressed. That method should be fully developed, tested, and validated. This is the reference point for evaluating any form of automation (Figure 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Items to be considered in automating systems\n\n\n\nThe documentation for the chosen method should include the bulleted list of items from Figure 1, as they describe the science aspects of the method. The last four points are important. The method should be validated since the manual procedure is a reference point for determining if the automated system is producing useful results. The reproducibility metric offers a means of evaluating at least one expected improvement in an automated system; you\u2019d expect less variability in the results. This requires a set of reference sample materials that can be repeatedly evaluated to compare the manual and automated systems, and to periodically test the methods in use to ensure that there aren\u2019t any trends developing that would compromise the method\u2019s use. Basically, this amounts to statistical quality control on the processes.\nThe next step is to decide what improvements you are looking for in an automated system: increased throughput, lower cost of operation, the ability to off-load human work, reduced variability, etc. In short, what are your goals?\nThat brings us to the matter of project planning. We\u2019re not going to go into a lot of depth in this piece about project planning, as there are a number of references[e] on the subject, including material produced by the former Institute for Laboratory Automation.[f] There are some aspects of the subject that we do need to touch on, however, and they include:\n\n justifying the project and setting expectations and goals;\n analyzing the process;\n scheduling automation projects; and\n budgeting.\nJustification, expectations, and goals \nBasically why are you doing this, what do you expect to gain? What arguments are you going to use to justify the work and expense involved in the project? How will you determine if the project is successful?\nFundamentally, automation efforts are about productivity and the bulleted items noted in the introduction of this piece, repeated below with additional commentary:\n\n Lower costs per test, and better control over expenditure: These can result from a reduction in labor and materials costs, including more predictable and consistent reagent usage per test.\n Stronger basis for better workflow planning: Informatics systems can provide better management over workloads and resource allocation, while key performance indicators can show where bottlenecks are occurring or if samples are taking too long to process. These can be triggers for procedure automation to improve throughput.\n Reproducibility: The test results from automated procedures can be expected to be more reproducible by eliminating the variability that is typical of steps executed by people. Small variation in dispensing reagents, for example, could be eliminated.\n Predictability: The time to completion for a given test is more predictable in automated programs; once the process starts it keeps going without interruptions that can be found in human centered activities\n Tighter adherence to procedures: Automated procedures have no choice but to be consistent in procedure execution; that is what programming and automation is about.\nOf these, which are important to your project? If you achieved these goals, what would it mean to your labs operations and the organization as a whole; this is part of the justification for carrying out the projects.\nAs noted earlier, there are several things to consider in order to justify a project. First, there has to be a growing need that supports a procedures automation, one that can\u2019t be satisfied by other means that could include adding people, equipment, and lab space, or outsourcing the work (with the added burden of insuring data quality and integrity, and integrating that work with the lab\u2019s data\/information). Second, the cost of the project must be balanced by it\u2019s benefits. This includes any savings in cost, people (not reducing headcount, but avoiding new hires), material, and equipment, as well as the improvement of timeliness of results and overall lab operations. Third, when considering project justification, the automated process\u2019s useful lifetime has to be long enough to justify the development work. And finally, the process has to be stable so that you aren\u2019t in a constant re-development situation (this differs from periodic upgrades and performance improvements, EVOP in manufacturing terms). One common point of failure in projects is changes in underlying procedures; if the basic process model changes, you are trying to hit a moving target. That ruins schedules and causes budgets to inflate.\nThis may seem like a lot of things to think about for something that could be as simple as perhaps moving from manual pipettes to automatic units, but that just means the total effort to do the work will be small. However it is still important since it impacts data quality and integrity, and your ability to defend your results should they be challenged. And, by the way, the issue of automated pipettes isn\u2019t simple; there is a lot to consider in properly specifying and using these products.[g]\n\nAnalyzing the process \nAssuming that you have a well-described, thoroughly tested and validated procedure, that process has to be analyzed for optimization and suitability for automation. This is an end-to-end evaluation, not just a examination of isolated steps. This is an important point. Looking at a single step without taking into account the rest of the process may improve that portion of the process but have consequences elsewhere.\nTake a common example: working in a testing environment where samples are being submitted by outside groups (Figure 2).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Lab sample processing, initial data entry through results\n\n\n\nMost LIMS will permit sample submitters (with appropriate permissions) to enter the sample description information directly into the LIMS, reducing some of the clerical burden. Standardizing on sample containers, with barcodes, reduces the effort and cost in some aspects of sample handling. A barcode scanner could be used to scan samples as they arrive into the lab, letting the system know that they are ready to be tested.\nThat brings us to an evaluation of the process as a whole, as well as an examination of the individual steps in the procedure. As shown in Figure 1, automation can be done in one of two ways: automating the full process or automating individual steps. Your choice depends on several factors, not the least of which is your comfort level and confidence in adopting automation as a strategy for increasing productivity. For some, concentrating on improvements in individual steps is an attractive approach. The cost and risk may be lower and if a problem occurs you can always backup to a fully manual implementation until they are resolved.\nCare does have to be taken in choosing which steps to improve. From one perspective, you\u2019d want to do the step-wise implementation of automation as close to the end of the process as possible. The problem with doing it earlier is that you may create a backup in later stages of the process. Optimizing step 2, for example, doesn\u2019t do you much good if step 3 is overloaded and requires more people, or additional (possibly unplanned) automation to relieve a bottleneck there. In short, before you automate or improve a given step, you need to be sure that downstream processing can absorb the increase in materials flow. In addition, optimizing all the individual steps, one at time, doesn\u2019t necessarily add up to a well-designed full system automation. The transition between steps may not be as effective or efficient if the system were evaluated as a whole. If the end of the process is carried out by commercial instrumentation, the ability to absorb more work is easier since most of these systems are automated with computer data acquisition and processing, and many have auto-samplers available to accumulate samples that can be processed automatically. Some of those auto-samplers have built in robotics for common sample handling functions. If the workload builds, additional instruments can pick up the load, and equipment such as Baytek International\u2019s TurboTube[15] can accumulate sample vials in a common system and route them to individual instruments for processing.\nAnother consideration for partial automation is where the process is headed in the future. If the need for the process persists over a long period of time, will you eventually get to the point of needing to redo the automation to an integrated stream? If so, is it better to take the plunge early on instead of continually expending resources to upgrade it?\nOther considerations include the ability to re-purpose equipment. If a process isn\u2019t used full-time (a justification for partial automation) the same components may be used in improving other processes. Ideally, if you go the full-process automation route, you\u2019ll have sufficient sample throughput to keep it running for an extended period of time, and not have to start and stop the system as samples accumulate. A smoothly running slower automation process is better than a faster system that lies idle for significant periods of time, particularly since startup and shutdown procedures may diminish the operational cost savings in both equipment use and people\u2019s time.\nAll these points become part of both the technical justification and budget requirements.\nAnalyzing the process: Simulation and modeling\nSimulation and modeling have been part of science and engineering for decades, supported by ever-increasing powerful computing hardware and software. Continuous systems simulations have shown us the details of how machinery works, how chemical reactions occur, and how chromatographic systems and other instrumentation behaves.[16] There is another aspect to modeling and simulation that is appropriate here.\nDiscrete-events simulation (DES) is used to model and understand processes in business and manufacturing applications, evaluating the interactions between service providers and customers, for example. One application of DES is to determine the best way to distribute incoming customers to a limited number of servers, taking into account that not all customers have the same needs; some will tie up a service provider a lot longer than others, as represented by the classic bank teller line problem. That is one question that discrete systems can analyze. This form of simulation and modeling is appropriate to event-driven processes where the action is focused on discrete steps (like materials moving from one workstation to another) rather than as a continuous function of time (most naturally occurring systems fall into this category, e.g., heat flow and models using differential equations).\nThe processes in your lab can be described and analyzed via DES systems.[17][18][19] Those laboratory procedures are a sequence of steps, each having a precursor, variable duration, and following step until the end of the process is reached; this is basically the same as a manufacturing operation where modeling and simulation have been used successfully for decades. DES can be used to evaluate those processes and ask questions that can guide you on the best paths to take in applying automation technologies and solving productivity or throughput problems. For example:\n\n What happens if we tighten up the variability in a particular step; how will that affect the rest of the system?\n What happens at the extremes of the variability in process steps; does it create a situation where samples pile up?\n How much of a workload can the process handle before one step becomes saturated with work and the entire system backs up?\n Can you introduce an alternate path to process those samples and avoid problems (e.g., if samples are held for too long in one stage, do they deteriorate)?\n Can the output of several parallel slower procedures be merged into a feed stream for a common instrumental technique?\nIn complex procedures some steps may be sensitive to small delays, and DES can help test and uncover them. Note that setting up these models will require the collection of a lot of data about the processes and their timing, so this is not something to be taken casually.\nPrevious research[16][17][18][19] suggests only a few ideas where simulation can be effective, including one where an entire labs operation\u2019s was evaluated. Models that extensive can be used to not only look at procedures, but also the introduction of informatics systems. This may appear to be a significant undertaking, and it can be depending on the complexity of the lab processes. However, simple processes can be initially modeled on spreadsheets to see if more significant effort is justified. Operations research, of which DES is a part, has been usefully applied in production operations to increase throughput and improve ROI. It might be successfully applied to some routine production oriented lab work.\nMost lab processes are linear in their execution, one step following another, with the potential for loop-backs should problems be recognized with samples, reagents (e.g., being out-of-date, doesn\u2019t look right, need to obtain new materials), or equipment (e.g., not functioning properly, out of calibration, busy due to other work). On one level, the modeling of a manually implemented process should appear to be simple: each step takes a certain amount of time. If you add up the times, you have a picture of the process execution through time. However, the reality is quite different if you take into account problems (and their resolution) that can occur in each of those steps. The data collection used to model the procedure can change how that picture looks and your ability to improve it. By monitoring the process over a number of iterations, you can find out how much variation there is in the execution time for each step and whether or not the variation is a normal distribution or skewed (e.g., if one step is skewed, how does it impact others?).\nQuestions to ask about potential problems that could occur at each step include:\n\n How often do problems with reagents occur and how much of a delay does that create?\n Is instrumentation always in calibration (do you know?), are there operational problems with devices and their control systems (what are the ramifications?), are procedures delayed due to equipment being in use by someone else, and how long does it take to make changeovers in operating conditions?\n What happens to the samples; do they degrade over time? What impact does this have on the accuracy of results and their reproducibility?\n How often are workflows interrupted by the need to deal with high-priority samples, and what effect does it have on the processing of other samples?\nJust the collection of data can suggest useful improvements before there are any considerations for automation, and perhaps negating the need for it. The answer to a lab\u2019s productivity might be as simple as adding another instrument if that is a bottleneck. It might also suggest that an underutilized device might be more productive if sample preparation for different procedures workflows were organized differently. Underutilization might be a consequence of the amount of time needed to prepare the equipment for service: doing so for one sample might be disproportionately time consuming (and expensive) and cause other samples to wait until there were enough of them to justify the preparation. It could also suggest that some lab processes should be outsourced to groups that have a more consistent sample flow and turn-around time (TAT) for that technique. Some of these points are illustrated in Figures 3a and 3b below.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3a. Simplified process views versus some modeling considerations. Note that the total procedure execution time is affected by the variability in each step, plus equipment and material availability delays; these can change from one day to the next in manual implementations.\n\n\n\n\n\n\n\n\n\n\n\n\n Figure 3b. The execution times of each step include the variable execution times of potential issues that can occur in each stage. Note that because each factor has a different distribution curve, the total execution time has a much wider variability than the individual factors.\n\n\n\nHow does the simulation system work? Once you have all the data set up, the simulation runs thousands of times using random number generators to pick out variables in execution times for each component in each step. For example, if there is a one-in-ten chance a piece of equipment will be in use when needed, 10% of the runs will show that with each one picking a delay time based on the input delay distribution function. With a large number of runs, you can see where delays exist and how they impact the overall processes behavior. You can also adjust the factors (what happens if equipment delays are cut in half) and see the effect of doing that. By testing the system, you can make better judgments on how to apply your resources.\nSome of the issues that surface may be things that lab personnel know about and just deal with. It isn\u2019t until the problems are looked at that the impact on operations are fully realized and addressed. Modeling and simulation may appear to be overkill for lab process automation, something reserved for large- scale production projects. The physical size of the project is not the key factor, it is the complexity of the system that matters and the potential for optimization.\nOne benefit of a well-structured simulation of lab processes is that it would provide a solid basis for making recommendations for project approval and budgeting. The most significant element in modeling and simulation is the initial data collection, asking lab personnel to record the time it takes to carry out steps. This isn\u2019t likely to be popular if they don\u2019t understand why it is being done and what the benefits will be to them and the lab; accurate information is essential. This is another case where \u201cbad data is worse than no data.\u201d\nGuidleines for process automation\nThere are two types of guidelines that will be of interest to those conducting automation work: those that help you figure out what to do and how to do it, and those that must be met to satisfy regulatory requirements (both those evaluated by internal or external groups or organizations).\nThe first is going to depend on the nature of the science and automation being done to support it. Equipment vendor community support groups can be of assistance. Additionally, professional groups like the Pharmaceutical Research and Manufacturers of America (PhRMA), International Society for Pharmaceutical Engineering (ISPE), and Parenteral Drug Association (PDA) in the pharmaceutical and biotechnology industrues, with similar organizations in other industries and other countries. This may seem like a large jump from laboratory work, but it is appropriate when we consider the ramification of full-process automation. You are essentially developing a manufacturing operation on a lab bench, and the same concerns that large-scale production have also apply here; you have to ensure that the process is maintained and in control. The same is true of manual or semi-automated lab work, but it is more critical in fully-automated systems because of the potential high volume of results that can be produced.\nThe second set is going to consist of regulatory guidelines from groups appropriate to your industry: the Food and Drug Administration (FDA), Environmental Protection Agency (EPA), and International Organization for Standardization (ISO), as well as international groups (e.g., GAMP, GALP) etc. The interesting point is that we are looking at a potentially complete automation scheme for a procedure; does that come under manufacturing or laboratory? The likelihood is that laboratory guidelines will apply since the work is being done within the lab's footprint; however, there are things that can be learned from their manufacturing counterparts that may assist in project management and documentation. One interesting consideration is what happens when fully automated testing, such as on-line analyzers, becomes integrated with both the lab and production or process control data\/information streams. Which regulatory guidelines apply? It may come down to who is responsible for managing and supporting those systems.\n\nScheduling automation projects \nThere are two parts to the schedule issue: how long is it going to take to compete the project (dependent on the process and people), and when do you start? The second point will be addressed here.\nThe timing of an automated process coming online is important. If it comes on too soon, there may not be enough work to justify it\u2019s use, and startup\/shutdown procedures may create more work than the system saves. If it comes too late, people will be frustrated with a heavy workload while the system that was supposed to provide relief is under development. \nIn Figure 4, the blue line represents the growing need for sample\/material processing using a given laboratory procedure. Ideally, you\u2019d like the automated version to be available when that blue line crosses the \u201cautomation needed on-line\u201d level of processing requirements; this the point where the current (manual?) implementation can no longer meet the demands of sample throughput requirements.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Timing the development of an automated system\n\n\n\nThose throughput limits are something you are going to have to evaluate and measure on a regular basis and use to make adjustments to the planning process (accelerating or slowing it as appropriate). How fast is the demand growing and at what point will your current methods be overwhelmed? Hiring more people is one option, but then the lab's operating expenses increase due to the cost of people, equipment, and lab space.\nOnce we have an idea of when something has to be working, we can begin the process of planning; note: the planning can begin at any point, it would be good to get the preliminaries done as soon as a manual process is finalized so that you have an idea of what you\u2019ll be getting into. Those preliminaries include looking at equipment that might be used (keeping track of its development), training requirements, developer resources, and implementation strategies, all of which would be updated as new information becomes available. The \u201cwe\u2019ll-get-to-it-when-we-need-it\u201d approach is just going to create a lot of stress and frustration.\nYou need to put together a first-pass project plan so that you can detail what you know, and more importantly what you don\u2019t know. The goal is to have enough information, updated as noted above, so that you can determine if an automated solution is feasible, make an informed initial choice between full and partial automation, and have a timeline for implementation. Any time estimate is going to be subject to change as you gather information and refine your implementation approach. The point of the timeline is to figure out how long the yellow box in Figure 4 is because that is going to tell you how much time you have to get the plan together and working; it is a matter of setting priorities and recognizing what they are. The time between now and the start of the yellow box is what you have to work with for planning and evaluating plans, and any decisions that are needed before you begin, including corporate project management requirements and approvals.\nThose plans have to include time for validation and the evaluation of the new implementation against the standard implementation. Does it work? Do we know how to use and maintain it? And are people educated in its use? Is there documentation for the project?\n\nBudgeting \nAt some point, all the material above and following this section comes down to budgeting: how much will it cost to implement a program and is it worth it? Of the two points, the latter is the one that is most important. How do you go about that? (Note: Some of this material is also covered in the webinar series A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work in the section on ROI.)\nWhat a lot of this comes down to is explaining and justifying the choices you\u2019ve made in your project proposal. We\u2019re not going to go into a lot of depth, but just note some of the key issues:\n\n Did you choose full or partial automation for your process?\n What drove that choice? If in your view it would be less expensive than the full automation of a process, how long will it be until the next upgrade is needed to another stage?\n How independent are the potential, sequential implementation efforts that may be undertaken in the future? Will there be a need to connect them, and if so, how will the incremental costs compare to just doing it once and getting it over with?\nThere is a tendency in lab work to treat problems and the products that might be used to address them in isolation. You see the need for a LIMS or ELN, or an instrument data system, and the focus is on those issues. Effective decisions have to consider both the immediate and longer-term aspects of a problem. If you want to get access to a LIMS, have you considered how it will affect other aspects of lab work such as connecting instrument to it?\nThe same holds true for partial automation as a solution to a lab process productivity problem. While you are addressing a particular step, should you be looking at the potential for synergism by addressing other concerns. Modeling and simulations of processes can help resolve that issue.\nHave you factored in the cost of support and education? The support issue needs to address the needs of lab personnel in managing the equipment and the options for vendor support, as well as the impact on IT groups. Note that the IT group will require access to vendor support, as well as being educated on their role in any project work.\nWhat happens if you don\u2019t automate? One way to justify the cost of a project is to help people understand what the lab\u2019s operations will be like without it. Will more people, equipment, space, or added shifts be needed? At what cost? What would the impact be on those who need the results and how would it affect their programs?\n\nBuild, buy, or cooperate? \nIn this write up and some of the referenced materials, we\u2019ve noted several times the benefits that clinical labs have gained through automation, although crediting it all to the use of automation alone isn\u2019t fair. What the clinical laboratory industry did was recognize that there was a need for the use of automation to solve problems with the operational costs of running labs, and recognition that they could benefit further by coming together and cooperatively addressing lab operational problems.\nIt\u2019s that latter point that made the difference and resulted in standardized communications, and purpose-built commercial equipment that could be used to implement automation in their labs. They also had common sample types, common procedures, and data processing. That same commonality applies to segments of industrial and academic lab work. Take life sciences as an example. Where possible, that industry has standardized on micro-plates for sample processing. The result is a wide selection of instruments and robotics built around that sample-holding format that greatly improves lab economics and throughput. While it isn\u2019t the answer to everything, it\u2019s a good answer to a lot of things.\nIf your industry segment came together and recognized that you used common procedures, how would you benefit by creating a common approach to automation instead of each lab doing it on their own? It would open the development of common products or product variations from vendors and relieve the need for each lab developing its own answer to the need. The result could be more effective and easily supportable solutions.\n\nProject planning \nOnce you\u2019ve decided on the project you are going to undertake, the next stage is looking at the steps needed to manage your project (Figure 5).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Steps in a laboratory automation project. This diagram is modeled after the GAMP V for systems validation.\n\n\n\nThe planning begins with the method description from Figure 1, which describes the science behind the project and the specification of how the automation is expected to be put into effect: as full-process automation, a specific step, or steps in the process. The provider of those documents is considered the \u201ccustomer\u201d and is consistent with GAMP V nomenclature (Figure 6); that consistency is important due to the need for system-wide validation protocols.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. GAMP V model for showing customer and supplier roles in specifying and evaluating project components for computer hardware and software.\n\n\n\nFrom there the \u201csupplier\u201d (e.g., internal development group, consultant, IT services, etc.) responds with a functional specification that is reviewed by the customer. The \u201canalysis, prototyping, and evaluation\u201d step, represented in the third box of Figure 5, is not the same as the process analysis noted earlier in this piece. The earlier section was to help you determine what work needed to be done and documented in the user requirements specification. The analysis and associated tasks here are specific to the implementation of this project. The colored arrows refer to the diagram in Figure 7. That process defines the equipment needed, dependencies, and options\/technologies for automation implementations, including robotics, instrument design requirements, pre-built automation (e.g., titrators, etc.) and any custom components. The documentation and specifications are part of the validation protocol.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 7. Defining dependencies and qualification of equipment\n\n\n\nThe prototyping function is an important part of the overall process. It is rare that someone will look at a project and come up with a working solution on the first pass. There is always tinkering and modifications that occur as you move from a blank slate to a working system. You make notes along the way about what should be done differently in the final product, and places where improvements or adjustments are needed. These all become part of the input to the system design specification that will be reviewed and approved by the customer and supplier. The prototype can be considered a proof of concept or a demonstration of what will occur in the finished product. Remember also that prototypes would not have to be validated since they wouldn\u2019t be used in a production environment; they are simply a test bed used prior to the development of a production system.\nThe component design specifications are the refined requirement for elements that will be used in the final design. Those refinements could point to updated models of components or equipment used, modifications needed, or recommendations for products with capabilities other than those used in the prototype.\nThe boxes on the left side of Figure 5 are documents that go into increasing depth as the system is designed and specified. The details in those items will vary with the extent of the project. The right side of the diagram is a series of increasingly sophisticated testing and evaluation against steps in the right side, culminating in the final demonstration that the system works, has been validated, and is accepted by the customer. It also means that lab and support personnel are educated in their roles.\n\nConclusions (so far) \n\u201cLaboratory automation\u201d has to give way to \u201claboratory automation engineering.\u201d From the initial need to the completion of the validation process, we have to plan, design, and implement successful systems on a routine basis. Just as the manufacturing industries transitioned from cottage industries to production lines and then to integrated production-information systems, the execution of laboratory science has to tread a similar path if the demands for laboratory results are going to be met in a financially responsible manner. The science is fundamental; however, we need to pay attention now to efficient execution.\n\nAbbreviations, acronyms, and initialisms \nAI: Artificial intelligence\nAuI: Augmented intelligence\nDES: Discrete-events simulation\nELN: Electronic laboratory notebook\nEPA: Environmental Protection Agency\nFDA: Food and Drug Administration\nFRB: Fast radio bursts\nGALP: Good automated laboratory practices\nGAMP: Good automated manufacturing practice\nISO: International Organization for Standardization\nLES: Laboratory execution system\nLIMS: Laboratory information management system\nML: Machine learning\nROI: Return on investment\nSDMS: Scientific data management system\nTAT: Turn-around time\n\r\n\n\nFootnotes \n\n\n\u2191 The term \"scientific manufacturing\" was first mentioned to the author by Mr. Alberto Correia, then of Cambridge Biomedical, Boston, MA. \n\n\u2191 Intelligent enterprise technologies referenced in the report include robotic process automation, machine learning, artificial intelligence, the internet Of things, predictive analysis, and cognitive computing. \n\n\u2191 Doug Engelbart found the field of human-computer interaction and is credited with the invention of the computer mouse, and the \u201cMother of All Demos\u201d in 1968. \n\n\u2191 See Metropolis (1927 film) on Wikipedia. \n\n\u2191 See for example https:\/\/www.projectmanager.com\/project-planning; the simplest thing to do it put \u201cproject planning\u201d in a search engine and browse the results for something interesting. \n\n\u2191 See for example https:\/\/theinformationdrivenlaboratory.wordpress.com\/category\/resources\/; note that any references to the ILA should be ignored as the original site is gone, with the domain name perhaps having been leased by another organization that has no affiliation with the original Institute for Laboratory Automation. \n\n\u2191 As a starting point, view the Artel, Inc. site as one source. Also, John Bradshaw gave an informative presentation on \u201cThe Importance of Liquid Handling Details and Their Impact on your Assays\u201d at the 2012 European Lab Automation Conference, Hamburg, Germany. \n\n\nAbout the author \nInitially educated as a chemist, author Joe Liscouski (joe dot liscouski at gmail dot com) is an experienced laboratory automation\/computing professional with over forty years of experience in the field, including the design and development of automation systems (both custom and commercial systems), LIMS, robotics and data interchange standards. He also consults on the use of computing in laboratory work. He has held symposia on validation and presented technical material and short courses on laboratory automation and computing in the U.S., Europe, and Japan. He has worked\/consulted in pharmaceutical, biotech, polymer, medical, and government laboratories. His current work centers on working with companies to establish planning programs for lab systems, developing effective support groups, and helping people with the application of automation and information technologies in research and quality control environments.\n\nReferences \n\n\n\u2191 Frey, C.B.; Osborne, M.A. (17 September 2013). \"The Future of Employment: How Susceptible Are Jobs to Computerisation?\" (PDF). Oxford Martin School, University of Oxford. https:\/\/www.oxfordmartin.ox.ac.uk\/downloads\/academic\/The_Future_of_Employment.pdf . Retrieved 04 February 2021 .   \n\n\u2191 Hsu, J. (24 September 2018). \"Is it aliens? Scientists detect more mysterious radio signals from distant galaxy\". NBC News MACH. https:\/\/www.nbcnews.com\/mach\/science\/it-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586 . Retrieved 04 February 2021 .   \n\n\u2191 Timmer, J. (18 July 2018). \"AI plus a chemistry robot finds all the reactions that will work\". Ars Technica. https:\/\/arstechnica.com\/science\/2018\/07\/ai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work\/5\/ . Retrieved 04 February 2021 .   \n\n\u2191 \"HelixAI - Voice Powered Digital Laboratory Assistants for Scientific Laboratories\". HelixAI. http:\/\/www.askhelix.io\/ . Retrieved 04 February 2021 .   \n\n\u2191 PharmaIQ News (20 August 2018). \"Automation, IoT and the future of smarter research environments\". PharmaIQ. https:\/\/www.pharma-iq.com\/pre-clinical-discovery-and-development\/news\/automation-iot-and-the-future-of-smarter-research-environments . Retrieved 04 February 2021 .   \n\n\u2191 6.0 6.1 PharmaIQ (14 November 2017). \"The Future of Drug Discovery: AI 2020\". PharmaIQ. https:\/\/www.pharma-iq.com\/pre-clinical-discovery-and-development\/whitepapers\/the-future-of-drug-discovery-ai-2020 . Retrieved 04 February 2021 .   \n\n\u2191 Rossetto, L. (2018). \"Fight the Dour\". Wired (October): 826\u20137. https:\/\/www.magzter.com\/stories\/Science\/WIRED\/Fight-The-Dour .   \n\n\u2191 \"OPUS Package: SEARCH & IDENT\". Bruker Corporation. https:\/\/www.bruker.com\/en\/products-and-solutions\/infrared-and-raman\/opus-spectroscopy-software\/search-identify.html . Retrieved 04 February 2021 .   \n\n\u2191 Bourne, D. (2013). \"My Boss the Robot\". Scientific American 308 (5): 38\u201341. doi:10.1038\/scientificamerican0513-38. PMID 23627215.   \n\n\u2191 SalesForce Research (2017). \"Second Annual State of IT\" (PDF). SalesForce. https:\/\/a.sfdcstatic.com\/content\/dam\/www\/ocms\/assets\/pdf\/misc\/2017-state-of-it-report-salesforce.pdf . Retrieved 04 February 2021 .   \n\n\u2191 \"Seal Analytical - Products\". Seal Analytical. https:\/\/seal-analytical.com\/Products\/tabid\/55\/language\/en-US\/Default.aspx . Retrieved 04 February 2021 .   \n\n\u2191 \"Bran+Luebbe\". SPX FLOW, Inc. https:\/\/www.spxflow.com\/bran-luebbe\/ . Retrieved 04 February 2021 .   \n\n\u2191 \"Array Tape Advanced Consumable\". Douglas Scientific. https:\/\/www.douglasscientific.com\/Products\/ArrayTape.aspx . Retrieved 04 February 2021 .   \n\n\u2191 \"Agilent 1200 Series Standard and Preparative Autosamplers - User Manual\" (PDF). Agilent Technologies. November 2008. https:\/\/www.agilent.com\/cs\/library\/usermanuals\/Public\/G1329-90012_StandPrepSamplers_ebook.pdf . Retrieved 04 February 2021 .   \n\n\u2191 \"iPRO Interface - Products\". Baytek International, Inc. https:\/\/www.baytekinternational.com\/products\/ipro-interface\/89-products . Retrieved 05 February 2021 .   \n\n\u2191 16.0 16.1 Joyce, J. (2018). \"Computer Modeling and Simulation\". Lab Manager (9): 32\u201335. https:\/\/www.labmanager.com\/laboratory-technology\/computer-modeling-and-simulation-1826 .   \n\n\u2191 17.0 17.1 Costigliola, A.; Ata\u00edde, F.A.P.; Vieira, S.M. et al. (2017). \"Simulation Model of a Quality Control Laboratory in Pharmaceutical Industry\". IFAC-PapersOnLine 50 (1): 9014-9019. doi:10.1016\/j.ifacol.2017.08.1582.   \n\n\u2191 18.0 18.1 Meng, L.; Liu, R.; Essick, C. et al. (2013). \"Improving Medical Laboratory Operations via Discrete-event Simulation\". Proceedings of the 2013 INFORMS Healthcare Conference. https:\/\/www.researchgate.net\/publication\/263238201_Improving_Medical_Laboratory_Operations_via_Discrete-event_Simulation .   \n\n\u2191 19.0 19.1 \"Application of discrete-event simulation in health care clinics: A survey\". Journal of the Operational Research Society 50: 109\u201323. 1999. doi:10.1057\/palgrave.jors.2600669.   \n\n\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/LII:Considerations_in_the_Automation_of_Laboratory_Procedures\">https:\/\/www.limswiki.org\/index.php\/LII:Considerations_in_the_Automation_of_Laboratory_Procedures<\/a>\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tLII\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 18 February 2021, at 19:46.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 888 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","e0147011cc1eb892e1a35e821657a6d9_html":"<body class=\"mediawiki ltr sitedir-ltr ns-202 ns-subject page-LII_Considerations_in_the_Automation_of_Laboratory_Procedures skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">LII:Considerations in the Automation of Laboratory Procedures<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\"><p><b>Title<\/b>: <i>Considerations in the Automation of Laboratory Procedures<\/i>\n<\/p><p><b>Author for citation<\/b>: Joe Liscouski, with editorial modifications by Shawn Douglas\n<\/p><p><b>License for content<\/b>: <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/creativecommons.org\/licenses\/by\/4.0\/\" target=\"_blank\">Creative Commons Attribution 4.0 International<\/a>\n<\/p><p><b>Publication date<\/b>: January 2021\n<\/p>\n\n\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Scientists have been dealing with the issue of <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_automation\" title=\"Laboratory automation\" class=\"wiki-link\" data-key=\"0061880849aeaca05f8aa27ae171f331\">laboratory automation<\/a> for decades, and during that time the meaning of those words has expanded from the basics of connecting an instrument to a computer, to the possibility of a fully integrated <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics_(academic_field)\" title=\"Informatics (academic field)\" class=\"wiki-link\" data-key=\"0391318826a5d9f9a1a1bcc88394739f\">informatics<\/a> infrastructure beginning with <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">sample<\/a> preparation and continuing on to the <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management system<\/a> (LIMS), <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_laboratory_notebook\" title=\"Electronic laboratory notebook\" class=\"wiki-link\" data-key=\"a9fbbd5e0807980106763fab31f1e72f\">electronic laboratory notebook<\/a> (ELN), and beyond. Throughout this evolution there has been one underlying concern: how do we go about doing this?\n<\/p><p>The answer to that question has changed from a focus on hardware and programming, to today\u2019s need for a lab-wide informatics strategy. We\u2019ve moved from the bits and bytes of assembly language programming to managing terabytes of files and data structures.\n<\/p><p>The high-end of the problem\u2014the large informatics database systems\u2014has received significant industry-wide attention in the last decade. The stuff on the lab bench, while the target of a lot of individual products, has been less organized and more experimental. Failed or incompletely met promises have to yield to planned successes. How we do it needs to change. This document is about the considerations required when making that change. The haphazard \"let's try this\" method has to give way to more engineered solutions and a realistic appraisal of the human issues, as well as the underlying technology management and planning.\n<\/p><p>Why is this important? Whether you are conducting intense laboratory experiments to produce data and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> or making chocolate chip cookies in the kitchen, two things remain important: productivity and the quality of the products. In either case, if the productivity isn\u2019t high enough, you won\u2019t be able to justify your work; if the quality isn\u2019t there, no one will want what you produce. Conducting laboratory work and making cookies have a lot in common. Your laboratories exist to answer questions. What happens if I do this? What is the purity of this material? What is the structure of this compound? The field of laboratories asking these questions is extensive, basically covering the entire array of lab bench and scientific work, including chemistry, life sciences, physics, and electronics labs. The more efficiently we answer those questions, the more likely it will be that these labs will continue operating and, that you\u2019ll achieve the goals your organization has set. At some point, it comes down to performance against goals and the return on the investment organizations make in lab operations.\n<\/p><p>In addition to product quality and productivity, there are a number of other points that favor automation over manual implementations of lab processes. They include:\n<\/p>\n<ul><li> lower costs per test;<\/li>\n<li> better control over expenditures;<\/li>\n<li> a stronger basis for better <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a> planning;<\/li>\n<li> reproducibility;<\/li>\n<li> predictably; and<\/li>\n<li> tighter adherence to procedures, i.e., consistency.<\/li><\/ul>\n<p>Lists similar to the one above can be found in justifications for lab automation, and cookie production, without further comment. It\u2019s just assumed that everyone agrees and that the reasoning is obvious. Since we are going to use those items to justify the cost and effort that goes into automation, we should take a closer look at them.\n<\/p><p>Lets begin with reproducibility, predictability, and consistency, very similar concerns that reflect automation\u2019s ability to produce the same product with the desired characteristics over and over. For data and information, that means that the same analysis on the same materials will yield the same results, that all the steps are documented and that the process is under control. The variability that creeps into the execution of a process by people is eliminated. That variability in human labor can result from the quality of training, equipment setup and calibration, readings from analog devices (e.g., meters, pipette meniscus, charts, etc.), there is a long list of potential issues.\n<\/p><p>Concerns with reproducibility, predictability, and consistency are common to production environments, general lab work, manufacturing, and even food service. There are several pizza restaurants in our area using one of two methods of making the pies. Both start the preparation the same way, spreading dough and adding cheese and toppings, but the differences are in how they are cooked. Once method uses standard ovens (e.g., gas, wood, or electric heating); the pizza goes in, the cook watches it, and then removes it when the cooking is completed. This leads to a lot of variability in the product, some a function of the cook\u2019s attention, some depending on requests for over or under cooking the crust. Some is based on \"have it your way\" customization. The second method uses a metal conveyor belt to move the pie through an oven. The oven temperature is set as is the speed of the belt, and as long as the settings are the same, you get a reproducible, consistent product order after order. It\u2019s a matter of priorities. Manual verses automated. Consistent product quality verses how the cook feels that day. In the end, reducing variability and being able to demonstrate consistent, accurate, results gives people confidence in your product.\n<\/p><p>Lower costs per test, better control over expenditures, and better workflow planning also benefit from automation. Automated processes are more cost-efficient since the sample throughput is higher and the labor cost is reduced. The cost per test and the material usage is predictable since variability in components used in testing is reduced or eliminated, and workflow planning is improved since the time per test is known, work can be better scheduled. Additionally, process scale-up should be easier if there is a high demand for particular procedures. However there is a lot of work that has to be considered before automation is realizable, and that is where this discussion is headed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"How_does_this_discussion_relate_to_previous_work.3F\">How does this discussion relate to previous work?<\/span><\/h2>\n<p>This work follows on the heels of two previous works:\n<\/p>\n<ul><li> <i><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.pda.org\/bookstore\/product-detail\/2684-computerized-systems-in-modern-lab\" target=\"_blank\">Computerized Systems in the Modern Laboratory: A Practical Guide<\/a><\/i> (2015): This book presents the range of informatics technologies, their relationship to each other, and the role they play in laboratory work. It differentiates a LIMS from an ELN and <a href=\"https:\/\/www.limswiki.org\/index.php\/Scientific_data_management_system\" title=\"Scientific data management system\" class=\"wiki-link\" data-key=\"9f38d322b743f578fef487b6f3d7c253\">scientific data management system<\/a> (SDMS) for example, contrasting their use and how they would function in different lab working environments. In addition, it covers topics such as support and regulatory issues.<\/li><\/ul>\n<ul><li> <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:A_Guide_for_Management:_Successfully_Applying_Laboratory_Systems_to_Your_Organization%27s_Work\" title=\"LII:A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work\" class=\"wiki-link\" data-key=\"00b300565027cb0518bcb0410d6df360\">A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work<\/a><\/i> (2018): This webinar series complements the above text. It begins by introducing the major topics in informatics (e.g., LIMS, ELN, etc.) and then discusses their use from a strategic viewpoint. Where and how do you start planning? What is your return on investment? What should get implemented first, and then what are my options? The series then moves on to developing an <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">information management<\/a> strategy for the lab, taking into account budgets, support, ease of implementation, and the nature of your lab\u2019s work.<\/li><\/ul>\n<p>The material in this write-up picks up where the last part of the webinar series ends. The last session covers lab processes, amd this picks up that thread and goes into more depth concerning a basic issue: how do you move from manual methods to automated systems?\n<\/p><p>Productivity has always been an issue in laboratory work. Until the 1950s, a lab had little choice but to add more people if more work needed to be done. Since then, new technologies have afforded wider options, including new instrument technologies. The execution of the work was still done by people, but the tools were better. Now we have other options. We just have to figure out when, if, and how to use them.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Before_we_get_too_far_into_this...\">Before we get too far into this...<\/span><\/h3>\n<p>With elements such as productivity, return on investment (ROI), <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_quality\" title=\"Data quality\" class=\"wiki-link\" data-key=\"7fe43b05eae4dfa9b5c0547cc8cfcceb\">data quality<\/a>, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">data integrity<\/a> as driving factors in this work, you shouldn\u2019t be surprised if a lot of the material reads like a discussion of manufacturing methodologies; we\u2019ve already seen some examples. We are talking about scientific work, but the same things that drive the elements noted in labs have very close parallels in product manufacturing. The work we are describing here will be referenced as \"scientific manufacturing,\" manufacturing or production in support of scientific programs.<sup id=\"rdp-ebb-cite_ref-1\" class=\"reference\"><a href=\"#cite_note-1\">[a]<\/a><\/sup>\n<\/p><p>The key points of a productivity conversation in both lab and material production environments are almost exact overlays, the only significant difference is that the results of the efforts are data and information in one case, and a physical item you might sell in the other. Product quality and integrity are valued considerations in both. For scientists, this may require an adjustment to their perspectives when dealing with automation. On the plus side, the lessons learned in product manufacturing can be applied to lab bench work, making the path to implementation a bit easier while providing a framework for understanding what a successful automation effort looks like. People with backgrounds in product manufacturing can be a useful resource in the lab, with a bit of an adjustment in perspective on their part.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Transitioning_from_typical_lab_operations_to_automated_systems\">Transitioning from typical lab operations to automated systems<\/span><\/h2>\n<p>Transitioning a lab from its current state of operations to one that incorporates automation can raise a number of questions, and people\u2019s anxiety levels. There are several questions that should be considered to set expectations for automated systems and how they will impact jobs and the introduction of new technologies. They include:\n<\/p>\n<ul><li> What will happen to people\u2019s jobs as a result of automation?<\/li>\n<li> What is the role of <a href=\"https:\/\/www.limswiki.org\/index.php\/Artificial_intelligence\" title=\"Artificial intelligence\" class=\"wiki-link\" data-key=\"0c45a597361ca47e1cd8112af676276e\">artificial intelligence<\/a> (AI) and <a href=\"https:\/\/www.limswiki.org\/index.php\/Machine_learning\" title=\"Machine learning\" class=\"wiki-link\" data-key=\"79aab39cfa124c958cd1dbcab3dde122\">machine learning<\/a> (ML) in automation?<\/li>\n<li> Where do we find the resources to carry out automation projects\/programs?<\/li>\n<li> What equipment would we need for automated processes, and will it be different that what we currently have?<\/li>\n<li> What role does a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_execution_system\" title=\"Laboratory execution system\" class=\"wiki-link\" data-key=\"774bdcab852f4d09565f0486bfafc26a\">laboratory execution system<\/a> (LES) play in laboratory automation?<\/li>\n<li> How do we go about planning for automation?<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"What_will_happen_to_people.E2.80.99s_jobs_as_a_result_of_automation.3F\">What will happen to people\u2019s jobs as a result of automation?<\/span><\/h3>\n<p>Stories are appearing in print, online, and in television news reporting about the potential for automation to replace human effort in the labor force. It seems like it is an all-or-none situation, either people will continue working in their occupations or automation (e.g., mechanical, software, AI, etc.) will replace them. The storyline is people are expensive and automated work can be less costly in the long run. If commercial manufacturing is a guide, automation is a preferred option from both a productivity and an ROI perspective. In order to make the productivity gains from automation similar to those seen in commercial manufacturing, there are some basic requirements and conditions that have to be met:\n<\/p>\n<ul><li> The process has to be well documented and understood, down to the execution of each step without variation, while error detection and recovery have to be designed in.<\/li>\n<li> The process has to remain static and be expected to continue over enough execution cycles to make it economically attractive to design, build, and maintain.<\/li>\n<li> Automation-compatible equipment has to be available. Custom-built components are going to be expensive and could represent a barrier to successful implementation.<\/li>\n<li> There has to be a driving need to justify the cost of automation; economics, the volume of work that has to be addressed, working with hazardous materials, and lack of educated workers are just a few of the factors that would need to be considered.<\/li><\/ul>\n<p>There are places in laboratory work where production-scale automation has been successfully implemented; life sciences applications for processes based on microplate technologies are one example. When we look at the broad scope of lab work across disciplines, most lab processes don\u2019t lend themselves to that level of automation, at least not yet. We\u2019ll get into this in more detail later. But that brings us back to the starting point: what happens to people's jobs?\n<\/p><p>In the early stages of manufacturing automation, as well as fields such as mining where work was labor intensive and repetitive, people did lose jobs when new methods of production were introduced. That shift from a human workforce to automated task execution is expanding as system designers probe markets from retail to transportation.<sup id=\"rdp-ebb-cite_ref-FreyTheFuture13_2-0\" class=\"reference\"><a href=\"#cite_note-FreyTheFuture13-2\">[1]<\/a><\/sup> Lower skilled occupations gave way first, and we find ourselves facing automation efforts that are moving up the skills ladder, most recently is the potential for automated driving, a technology that has yet to be fully embraced but is moving in that direction. The problem that leaves us with is providing displaced workers with a means of employment that gives them at least a living income, and the purpose, dignity, and self-worth that they\u2019d like to have. This is going to require significant education, and people are going to have to come to grips with the realization that education never stops.\n<\/p><p>Due to the push for increased productivity, lab work has seen some similar developments in automation. The development of automated pipettes, titration stations, auto-injectors, computer-assisted instrumentation, and automation built to support microplate technologies represent just a few places where specific tasks have been addressed. However these developments haven\u2019t moved people out of the workplace as has happened in manufacturing, mining, etc. In some cases they\u2019ve changed the work, replacing repetitive time-consuming tasks with equipment that allows lab personnel to take on different tasks. In other cases the technology addresses work that couldn\u2019t be performed in a cost-effective manner with human effort; without automation, that work might just not be feasible due to the volume of work (whose delivery might be limited by the availability of the right people, equipment, and facilities) or the need to work with hazardous materials. Automation may prevent the need for hiring new people while giving those currently working more challenging tasks.\n<\/p><p>As noted in the previous paragraph, much of the automation in lab work is at the task level: equipment designed to carry out a specific function such as Karl-Fisher titrations. Some equipment designed around microplate formats can function at both the task level and as part of user-integrated robotics system. This gives the planner useful options about the introduction of automation that makes it easier for personnel to get accustomed to automation before moving into scientific manufacturing.\n<\/p><p>Overall, laboratory people shouldn\u2019t be loosing their jobs as a result of lab automation, but they do have to be open to changes in their jobs, and that could require an investment in their education. Take someone whose current job is to carry out a lab procedure, someone who understands all aspects of the work, including troubleshooting equipment, reagents, and any special problems that may crop up. Someone else may have developed the procedure, but that person is the expert in its execution.\n<\/p><p>First of all you need these experts to help plan and test the automated systems if you decide to create that project. These would also be the best people to educate as automated systems managers; they know how the process is supposed to work and should be in a position to detect problems. If it crashes, you\u2019ll need someone who can cover the work while problems are be addressed. Secondly, if lab personnel get the idea that they are watching their replacement being installed, they may leave before the automated systems are ready. In the event of a delay, you\u2019ll have a backlog and no one to handle it.\n<\/p><p>Beyond that, people will be freed from the routine of carrying out processes and be able to address work that had been put on a back burner until it could be addressed. As we move toward automated systems, jobs will change by expansion to accommodate typical lab work, as well as the management, planning, maintenance, and evolution of laboratory automation and computing.\n<\/p><p>Automation in lab work is not an \"all or none\" situation. Processes can be structured so that the routine work is done by systems, and the analyst can spend time reviewing the results, looking for anomalies and interesting patterns, while being able to make decisions about the need for and nature of follow-on efforts.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"What_is_the_role_of_AI_and_ML_in_automation.3F\">What is the role of AI and ML in automation?<\/span><\/h3>\n<p>When we discuss automation, what we are referencing now is basic robotics and programming. AI may, and likely will, play a role in the work, but first we have to get the foundations right before we consider the next step; we need to put in the human intelligence first. Part of the issue with AI is that we don\u2019t know what it is.\n<\/p><p>Science fiction aside, many of today's applications of AI have a limited role in lab work today. Here are some examples:\n<\/p>\n<ul><li> Having a system that can bring up all relevant information on a research question\u2014a sort of super Google\u2014or a variation of IBM\u2019s Watson could have significant benefits.<\/li>\n<li> Analyzing complex data or large volumes of data could be beneficial, e.g., the analysis of radio astronomy data to find fast radio bursts (FRB). After discovering 21 FRB signals upon analyzing five hours of data, researchers at Green Bank Telescope used AI to analyze 400 terabytes of older data and detected another 100.<sup id=\"rdp-ebb-cite_ref-HsuIsIt18_3-0\" class=\"reference\"><a href=\"#cite_note-HsuIsIt18-3\">[2]<\/a><\/sup><\/li>\n<li> \"[A] team at Glasgow University has paired a machine-learning system with a robot that can run and analyze its own chemical reaction. The result is a system that can figure out every reaction that's possible from a given set of starting materials.\"<sup id=\"rdp-ebb-cite_ref-TimmerAIPlus18_4-0\" class=\"reference\"><a href=\"#cite_note-TimmerAIPlus18-4\">[3]<\/a><\/sup><\/li>\n<li> HelixAI is using Amazon's Alexa as a digital assitant for laboratory work.<sup id=\"rdp-ebb-cite_ref-HelixAIHome_5-0\" class=\"reference\"><a href=\"#cite_note-HelixAIHome-5\">[4]<\/a><\/sup><\/li><\/ul>\n<p>Note that the points above are research-based applications, not routine production environments where regulatory issues are important. While there are research applications that might be more forgiving of AI systems because the results are evaluated by human intelligence, and problematic results can be made subject to further verification, data entry systems such as voice entry have to be carefully tested and the results of that data entry verified and shown to be correct.\n<\/p><p>Pharma IQ continues to publish material on advanced topics in laboratory informatics, including articles on how labs are benefiting from new technologies<sup id=\"rdp-ebb-cite_ref-PharmaIQNewsAutom18_6-0\" class=\"reference\"><a href=\"#cite_note-PharmaIQNewsAutom18-6\">[5]<\/a><\/sup> and survey reports such as <i>AI 2020: The Future of Drug Discovery<\/i>. In that report they note<sup id=\"rdp-ebb-cite_ref-PharmaIQTheFuture17_7-0\" class=\"reference\"><a href=\"#cite_note-PharmaIQTheFuture17-7\">[6]<\/a><\/sup>:\n<\/p>\n<ul><li> \"94% of pharma professionals expect that intelligent technologies will have a noticeable impact on the pharmaceutical industry over the next two years.\"<\/li>\n<li> \"Almost one fifth of pharma professionals believe that we are on the cusp of a revolution.\"<\/li>\n<li> \"Intelligent automation and predictive analytics are expected to have the most significant impact on the industry.\"<\/li>\n<li> \"However, a lack of understanding and awareness about the benefits of AI-led technologies remain a hindrance to their implementation.\"<\/li><\/ul>\n<p>Note that these are expectations, not a reflection of current reality. That same report makes comments about the impact of AI on headcount disruption, asking, \"Do you expect intelligent enterprise technologies<sup id=\"rdp-ebb-cite_ref-8\" class=\"reference\"><a href=\"#cite_note-8\">[b]<\/a><\/sup> to significantly cut and\/or create jobs in pharma through 2020?\" Among the responses, 47 percent said they expected those technologies to do both, 40 percent said it will create new job opportunities, and 13 percent said there will be no dramatic change, with zero percent saying they expected solely job losses.<sup id=\"rdp-ebb-cite_ref-PharmaIQTheFuture17_7-1\" class=\"reference\"><a href=\"#cite_note-PharmaIQTheFuture17-7\">[6]<\/a><\/sup>\n<\/p><p>While there are high levels of expectations and hopes for results, we need to approach the idea of AI in labs with some caution. We read about examples based on machine learning (ML), for example using computer systems to recognize cats in photos, to recognized faces in a crowd, etc. We don\u2019t know how they accomplish their tasks, and we can\u2019t analyze their algorithms and decision-making. That leaves us with testing in quality, which at best is an uncertain process with qualified results (it has worked so far). One problem with testing AI systems based on ML is that they are going to continually evolve, so testing may affect the ML processes by providing a bias. It may also cause continued, redundant testing, because something we thought was evaluated was changed by the \u201cexperiences\u201d the AI based it\u2019s learning on. As one example, could the AI modify the science through process changes without our knowing because it didn\u2019t understand the science or the goals of the work?\n<\/p><p>AI is a black box with ever-changing contents. That shouldn\u2019t be taken as a condemnation of AI in the lab, but rather as a challenge to human intelligence in evaluating, proving, and applying the technology. That application includes defining the operating boundaries of an AI system. Rather than creating a master AI for a complete process, we may elect to divide the AI\u2019s area of operation into multiple, independent segments, with segment integration occurring in later stages once we are confident in their ability to work and show clear evidence of systems stability. In all of this we need to remember that our goal is the production of high-quality data and information in a controlled, predictable environment, not gee-wiz technology. One place where AI (or clever programming) could be of use is in better workflow planning, which takes into account current workloads and assignments, factors in the inevitable panic-level testing need, and, perhaps in a QC\/production environment, anticipates changes in analysis requirements based on changes in production operations.\n<\/p><p>Throughout this section I've treated \u201cAI\u201d as \u201cartificial intelligence,\u201d its common meaning. There may be a better way of looking at it for lab use as, noted in this excerpt from the October 2018 issue of Wired magazine<sup id=\"rdp-ebb-cite_ref-RossettoFight18_9-0\" class=\"reference\"><a href=\"#cite_note-RossettoFight18-9\">[7]<\/a><\/sup>:\n<\/p>\n<blockquote>Augmented intelligence. Not \u201cartificial,\u201d but how Doug Engelbart<sup id=\"rdp-ebb-cite_ref-10\" class=\"reference\"><a href=\"#cite_note-10\">[c]<\/a><\/sup> envisioned our relationship with computer: AI doesn\u2019t replace humans. It offers idiot-savant assistants that enable us to become the best humans we can be.<\/blockquote>\n<p>Augmented intelligence (AuI) is a better term for what we might experience in lab work, at least in the near future. It suggests something that is both more realistic and attainable, with the synergism that would make it, and automation, attractive to lab management and personnel\u2014a tool they can work with and improve lab operations that doesn\u2019t carry the specter of something going on that they don\u2019t understand or control. OPUS\/SEARCH from Bruker might be just such an entry in this category.<sup id=\"rdp-ebb-cite_ref-BrukerOPUS_11-0\" class=\"reference\"><a href=\"#cite_note-BrukerOPUS-11\">[8]<\/a><\/sup> AuI may serve as a first-pass filter for large data sets\u2014as noted in the radio astronomy and chemistry examples noted earlier\u2014reducing those sets of data and information to smaller collections that human intelligence can\/should evaluate. However, that does put a burden on the AuI to avoid excessive false positives or negatives, something that can be adjusted over time.\n<\/p><p>Beyond that there is the possibility of more cooperative work between people and AuI systems. An article in <i>Scientific American<\/i> titled \u201cMy Boss the Robot\u201d<sup id=\"rdp-ebb-cite_ref-BourneMyBoss13_12-0\" class=\"reference\"><a href=\"#cite_note-BourneMyBoss13-12\">[9]<\/a><\/sup> describes the advantage of a human-robot team, with the robot doing the heavy work and the human\u2014under the robots guidance\u2014doing work he was more adept at, verses a team of experts with the same task. The task, welding a Humvee frame, was competed by the human machine pair in 10 hours at a cost of $1,150; the team of experts took 89 hours and a cost of $7,075. That might translate into terms of laboratory work by having a robot do routine, highly repetitive tasks and the analyst overseeing the operation and doing higher-level analysis of the results.\n<\/p><p>Certainly, AI\/AuI is going to change over time as programming and software technology becomes more sophisticated and capable; today\u2019s example of AuI might be seen as tomorrow\u2019s clever software. However, a lot depends on the experience of the user.\n<\/p><p>There is something important to ask about laboratory technology development, and AI in particular: is the direction of development going to be the result of someone\u2019s innovation that people look at and embrace, or will it be the result of a deliberate choice of lab people saying \u201cthis is where we need to go, build systems that will get us there\u201d? The difference is important, and lab managers and personnel need to be in control of the planning and implementation of systems.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Where_do_we_find_the_resources_to_carry_out_automation_projects.2Fprograms.3F\">Where do we find the resources to carry out automation projects\/programs?<\/span><\/h3>\n<p>Given the potential scope of work, you may need people with skills in programming, robotics, instrumentation, and possibly mechanical or electrical engineering if off-the-shelf components aren\u2019t available. The biggest need is for people who can do the planning and optimization that is needed as you move from manual to semi- or fully-automated systems, particularly specialists in process engineering who can organize and plan the work, including the process controls and provision for statistical process control.\n<\/p><p>We need to develop people who are well versed in laboratory work and the technologies that can be applied to that work, as assets in laboratory automation development and planning. In the past, this role has been filled with lab personnel having an interest in the subject, IT people willing to extend their responsibilities, and\/or outside consultants. A 2017 report by Salesforce Research states \"77% of IT leaders believe IT functions as an extension\/partner of business units rather than as a separate function.\"<sup id=\"rdp-ebb-cite_ref-SalesForceSecondAnn17_13-0\" class=\"reference\"><a href=\"#cite_note-SalesForceSecondAnn17-13\">[10]<\/a><\/sup> The report makes no mention of laboratory work or manufacturing aside from those being functions within businesses surveyed. Unless a particular effort is made, IT personnel rarely have the backgrounds needed to meet the needs of lab work. In many cases, they will try and fit lab needs into software they are already familiar with, rather then extend their backgrounds into new computational environments. Office and pure database applications are easily handled, but when we get to the lab bench, it's another matter entirely.\n<\/p><p>The field is getting complex enough that we need people whose responsibilities span both science and technology. This subject is discussed in the webinar series <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:A_Guide_for_Management:_Successfully_Applying_Laboratory_Systems_to_Your_Organization%27s_Work\" title=\"LII:A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work\" class=\"wiki-link\" data-key=\"00b300565027cb0518bcb0410d6df360\">A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work<\/a><\/i>, Part 5 \"Supporting Laboratory Systems.\"\n<\/p>\n<h3><span class=\"mw-headline\" id=\"What_equipment_would_we_need_for_automated_processes.2C_and_will_it_be_different_that_what_we_currently_have.3F\">What equipment would we need for automated processes, and will it be different that what we currently have?<\/span><\/h3>\n<p>This is an interesting issue and it directly addresses the commitment labs have to automation, particularly robotics. In the early days of lab automation when Zymark (Zymate and Benchmate), <a href=\"https:\/\/www.limswiki.org\/index.php\/PerkinElmer_Inc.\" title=\"PerkinElmer Inc.\" class=\"wiki-link\" data-key=\"dabda40785b60866d056709e611512f8\">Perkin Elmer<\/a>, and Hewlett Packard (ORCA) were the major players in the market, the robot had to adapt to equipment that was designed for human use: standard laboratory equipment. They did that through special modifications and the use of different grippers to handle test tubes, beakers, and flasks. While some companies wanted to test the use of robotics in the lab, they didn\u2019t want to invest in equipment that could only be used with robots; they wanted lab workers to pick up where the robots left off in case the robots didn\u2019t work.\n<\/p><p>Since then, equipment has evolved to support automation more directly. In some cases it is a device (e.g., a balance, pH meter, etc.) that has front panel human operator capability and rear connectors for computer communications. Liquid handling systems have seen the most advancement through the adoption of microplate formats and equipment designed to work with them. However, the key point is standardization of the sample containers. Vials and microplates lend themselves to a variety of automation devices, from sample processing to auto-injectors\/samplers. The issue is getting the samples into those formats.\n<\/p><p>One point that labs, in any scientific discipline, have to come to grips with is the commitment to automation. That commitment isn\u2019t going to be done on a lab-wide basis, but on a procedure-by-procedure basis. Full automation may not be appropriate for all lab work, whereas partial automation may be a better choice, and in some cases no automation may be required (we\u2019ll get into that later). The point that needs to be addressed is the choice of equipment. In most cases, equipment is designed for use by people, with options for automation and electronic communications. However, if you want to maximize throughput, you may have to follow examples from manufacturing and commit to equipment that is only used by automation. That will mean a redesign of the equipment, a shared risk for both the vendors and the users. The upside to this is that equipment can be specifically designed for a task, be more efficient, have the links needed for integration, use less material, and, more likely, take up less space. One example is the microplate, allowing for tens, hundreds, or thousands (depending on the plate used) of sample cells in a small space. What used to take many cubic feet of space as test tubes (the precursor to using microplates) is now a couple of cubic inches, using much less material and working space. Note, however, that while microplates are used by lab personnel, their use in automated systems provides greater efficiency and productivity.\n<\/p><p>The idea of equipment used only in an automated process isn\u2019t new. The development and commercialization of segmented flow analyzers\u2014initially by Technicon in the form of the AutoAnalyzers for general use, and the SMA (Sequential Multiple Analyzer) and SMAC (Sequential Multiple Analyzer with Computer) in clinical markets\u2014improved a lab's ability to process samples. These systems were phased out with new equipment that consumed less material. Products like these are being provided by Seal Analytical<sup id=\"rdp-ebb-cite_ref-SealAnal_14-0\" class=\"reference\"><a href=\"#cite_note-SealAnal-14\">[11]<\/a><\/sup> for environmental work and Bran+Luebbe (a division of SPX Process Equipment in Germany).<sup id=\"rdp-ebb-cite_ref-BranLuebbe_15-0\" class=\"reference\"><a href=\"#cite_note-BranLuebbe-15\">[12]<\/a><\/sup>\n<\/p><p>The issue in committing to automated equipment is that vendors and users will have to agree on equipment specifications and use them within procedures. One place this has been done successfully is in clinical chemistry labs. What other industry workflows could benefit? Do the vendors lead or do the users drive the issue? Vendors need to be convinced that there is a viable market for product before making an investment, and users need to be equally convinced that they will succeed in applying those products. In short, procedures that are important to a particular industry have to be identified, and both users and vendors have to come together to develop automated procedure and equipment specifications for products. This has been done successfully in clinical chemistry markets to the extent that equipment is marketed for use as validated for particular procedures.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"What_role_does_a_LES_play_in_laboratory_automation.3F\">What role does a LES play in laboratory automation?<\/span><\/h3>\n<p>Before ELNs settled into their current role in laboratory work, the initial implementations differed considerably from what we have now. LabTech Notebook was released in 1986 (discontinued in 2004) to provide communications between computers and devices that used RS-232 serial communications. In the early 2000s SmartLab from Velquest was the first commercial product to carry the \"electronic laboratory notebook\" identifier. That product became a stand-alone entry in the laboratory execution system (LES) market; since its release, the same conceptual functionality has been incorporated into LIMS and ELNs that fit the more current expectation for an ELN.\n<\/p><p>At it\u2019s core, LES are scripted test procedures that an analyst would follow to carry out a laboratory method, essentially functioning as the programmed execution of a lab process. Each step in a process is described, followed exactly, and provision is made within the script for data collection. In addition, the LES can\/will (depending on the implementation; \"can\" in the case of SmartLab) check to see if the analyst is qualified to carry out the work and that the equipment and reagents are current, calibrated, and suitable for use. The systems can also have access to help files that an analyst can reference if there are questions about how to carry out a step or resolve issues. Beyond that, the software had the ability to work with lab instruments and automatically acquire data either through direct interfaces (e.g., balances, pH meters, etc.) or through parsing PDF files of instrument reports.\n<\/p><p>There are two reasons that these systems are attractive. First, they provide for a rigorous execution of a process with each step being logged as it is done. Second, that log provides a regulatory inspector with documented evidence that the work was done properly, making it easier for the lab to meet any regulatory burden.\n<\/p><p>Since the initial development of SmartLab, that product has changed ownership and is currently in the hands of <a href=\"https:\/\/www.limswiki.org\/index.php\/Dassault_Syst%C3%A8mes_SA\" title=\"Dassault Syst\u00e8mes SA\" class=\"wiki-link\" data-key=\"1be69bd73e35bc3db0c3229284bf9416\">Dassault Syst\u00e8mes<\/a> as part of the BIOVIA product line. As noted above, LIMS and ELN vendors have incorporated similar functionality into their products. Using those features requires \u201cscripting\u201d (in reality, software development), but it does allow the ability to access the database structures within those products. The SmartLab software needed programmed interfaces to other vendors' LIMS and ELNs to gain access to the same information.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"What_does_this_have_to_do_with_automation.3F\">What does this have to do with automation?<\/span><\/h4>\n<p>When we think about automated systems, particularly full-automation with robotic support, it is a programmed process from start to finish. The samples are introduced at the start, and the process continues until the final data\/information is reported and stored. These can be large scale systems using microplate formats, including tape-based systems from Douglas Scientific<sup id=\"rdp-ebb-cite_ref-DouglasScientificArrayTape_16-0\" class=\"reference\"><a href=\"#cite_note-DouglasScientificArrayTape-16\">[13]<\/a><\/sup>, programmable autosamplers such as those from <a href=\"https:\/\/www.limswiki.org\/index.php\/Agilent_Technologies,_Inc.\" title=\"Agilent Technologies, Inc.\" class=\"wiki-link\" data-key=\"dcea1a676a012bcbe3af9562dd17f8a0\">Agilent<\/a><sup id=\"rdp-ebb-cite_ref-Agilent1200Series_17-0\" class=\"reference\"><a href=\"#cite_note-Agilent1200Series-17\">[14]<\/a><\/sup>, or systems built around robotics arms from a variety of vendors that move samples from one station to another.\n<\/p><p>Both LES and the automation noted in the previous paragraph have the following point in common: there is a strict process that must be followed, with no provision for variation. The difference is that in one case that process is implemented completely through the use of computers, as well as electronic and mechanical equipment. In the other case, the process is being carried out by lab personnel using computers, as well as electronic and mechanical lab equipment. In essence, people take the place of mechanical robots, which conjures up all kinds of images going back to the 1927 film <i>Metropolis<\/i>.<sup id=\"rdp-ebb-cite_ref-18\" class=\"reference\"><a href=\"#cite_note-18\">[d]<\/a><\/sup> Though the LES represents a step toward more sophisticated automation, both methods still require:\n<\/p>\n<ul><li> programming, including \u201cscripting\u201d (the LES methods are a script that has to be followed);<\/li>\n<li> validated, proven processes; and<\/li>\n<li> qualified staff, though the qualifications differ. (In both cases they have to be fully qualified to carry out the process in question. However in the full automation case, they will require more education on running, managing, and troubleshooting the systems.)<\/li><\/ul>\n<p>In the case of full automation, there has to be sufficient justification for the automation of the process, including sufficient sample processing. The LES-human implementation can be run for a single sample if needed, and the operating personnel can be trained on multiple procedures, switching tasks as needed. Electro-mechanical automation would require a change in programming, verification that the system is operating properly, and may require equipment re-configuration. Which method is better for a particular lab depends on trade-offs between sample load, throughput requirements, cost, and flexibility. People are adaptable, easily moving between tasks, whereas equipment has to be adapted to a task.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"How_do_we_go_about_planning_for_automation.3F\">How do we go about planning for automation?<\/span><\/h3>\n<p>There are three forms of automation to be considered:\n<\/p>\n<ol><li> No automation \u2013 Instead, the lab relies on lab personnel to carry out all steps of a procedure.<\/li>\n<li> Partial automation \u2013 Automated equipment is used to carry out steps in a procedure. Given the current state of laboratory systems, this is the most prevalent since most lab equipment has computer components in them to facilitate their use.<\/li>\n<li> Full automation - The entire process is automated. The definition of \u201centire\u201d is open to each labs interpretation and may vary from one process to another. For example, some samples may need some handing before they are suitable for use in a procedure. That might be a selection process from a freezer, grinding materials prior to a solvent extraction, and so on, representing cases where the equipment available isn\u2019t suitable for automated equipment interaction. One goal is to minimize this effort since it can put a limit on the productivity of the entire process. This is also an area where negotiation between the lab and the sample submitter can be useful. Take plastic pellets for example, which often need to be ground into a course powder before they can be analyzed; having the submitter provide them in this form will reduce the time and cost of the analysis. Standardizing on the sample container can also facilitate the analysis (having the lab provide the submitter with standard sample vials using barcodes or RFID chips can streamline the process).<\/li><\/ol>\n<p>One common point that these three forms share is a well-described method (procedure, process) that needs to be addressed. That method should be fully developed, tested, and validated. This is the reference point for evaluating any form of automation (Figure 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Liscouski_ConsidAutoLabProc21.png\" class=\"image wiki-link\" data-key=\"684c380f41ae1fa9a0e3ed8844cc8c6d\"><img alt=\"Fig1 Liscouski ConsidAutoLabProc21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/31\/Fig1_Liscouski_ConsidAutoLabProc21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Items to be considered in automating systems<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The documentation for the chosen method should include the bulleted list of items from Figure 1, as they describe the science aspects of the method. The last four points are important. The method should be validated since the manual procedure is a reference point for determining if the automated system is producing useful results. The reproducibility metric offers a means of evaluating at least one expected improvement in an automated system; you\u2019d expect less variability in the results. This requires a set of reference sample materials that can be repeatedly evaluated to compare the manual and automated systems, and to periodically test the methods in use to ensure that there aren\u2019t any trends developing that would compromise the method\u2019s use. Basically, this amounts to statistical quality control on the processes.\n<\/p><p>The next step is to decide what improvements you are looking for in an automated system: increased throughput, lower cost of operation, the ability to off-load human work, reduced variability, etc. In short, what are your goals?\n<\/p><p>That brings us to the matter of project planning. We\u2019re not going to go into a lot of depth in this piece about project planning, as there are a number of references<sup id=\"rdp-ebb-cite_ref-19\" class=\"reference\"><a href=\"#cite_note-19\">[e]<\/a><\/sup> on the subject, including material produced by the former Institute for Laboratory Automation.<sup id=\"rdp-ebb-cite_ref-20\" class=\"reference\"><a href=\"#cite_note-20\">[f]<\/a><\/sup> There are some aspects of the subject that we do need to touch on, however, and they include:\n<\/p>\n<ul><li> justifying the project and setting expectations and goals;<\/li>\n<li> analyzing the process;<\/li>\n<li> scheduling automation projects; and<\/li>\n<li> budgeting.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Justification.2C_expectations.2C_and_goals\">Justification, expectations, and goals<\/span><\/h4>\n<p>Basically why are you doing this, what do you expect to gain? What arguments are you going to use to justify the work and expense involved in the project? How will you determine if the project is successful?\n<\/p><p>Fundamentally, automation efforts are about productivity and the bulleted items noted in the introduction of this piece, repeated below with additional commentary:\n<\/p>\n<ul><li> Lower costs per test, and better control over expenditure: These can result from a reduction in labor and materials costs, including more predictable and consistent reagent usage per test.<\/li>\n<li> Stronger basis for better workflow planning: Informatics systems can provide better management over workloads and resource allocation, while key performance indicators can show where bottlenecks are occurring or if samples are taking too long to process. These can be triggers for procedure automation to improve throughput.<\/li>\n<li> Reproducibility: The test results from automated procedures can be expected to be more reproducible by eliminating the variability that is typical of steps executed by people. Small variation in dispensing reagents, for example, could be eliminated.<\/li>\n<li> Predictability: The time to completion for a given test is more predictable in automated programs; once the process starts it keeps going without interruptions that can be found in human centered activities<\/li>\n<li> Tighter adherence to procedures: Automated procedures have no choice but to be consistent in procedure execution; that is what programming and automation is about.<\/li><\/ul>\n<p>Of these, which are important to your project? If you achieved these goals, what would it mean to your labs operations and the organization as a whole; this is part of the justification for carrying out the projects.\n<\/p><p>As noted earlier, there are several things to consider in order to justify a project. First, there has to be a growing need that supports a procedures automation, one that can\u2019t be satisfied by other means that could include adding people, equipment, and lab space, or outsourcing the work (with the added burden of insuring data quality and integrity, and integrating that work with the lab\u2019s data\/information). Second, the cost of the project must be balanced by it\u2019s benefits. This includes any savings in cost, people (not reducing headcount, but avoiding new hires), material, and equipment, as well as the improvement of timeliness of results and overall lab operations. Third, when considering project justification, the automated process\u2019s useful lifetime has to be long enough to justify the development work. And finally, the process has to be stable so that you aren\u2019t in a constant re-development situation (this differs from periodic upgrades and performance improvements, EVOP in manufacturing terms). One common point of failure in projects is changes in underlying procedures; if the basic process model changes, you are trying to hit a moving target. That ruins schedules and causes budgets to inflate.\n<\/p><p>This may seem like a lot of things to think about for something that could be as simple as perhaps moving from manual pipettes to automatic units, but that just means the total effort to do the work will be small. However it is still important since it impacts data quality and integrity, and your ability to defend your results should they be challenged. And, by the way, the issue of automated pipettes isn\u2019t simple; there is a lot to consider in properly specifying and using these products.<sup id=\"rdp-ebb-cite_ref-21\" class=\"reference\"><a href=\"#cite_note-21\">[g]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Analyzing_the_process\">Analyzing the process<\/span><\/h4>\n<p>Assuming that you have a well-described, thoroughly tested and validated procedure, that process has to be analyzed for optimization and suitability for automation. This is an end-to-end evaluation, not just a examination of isolated steps. This is an important point. Looking at a single step without taking into account the rest of the process may improve that portion of the process but have consequences elsewhere.\n<\/p><p>Take a common example: working in a testing environment where samples are being submitted by outside groups (Figure 2).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Liscouski_ConsidAutoLabProc21.png\" class=\"image wiki-link\" data-key=\"e1944129f376f676452d5befa53e5a78\"><img alt=\"Fig2 Liscouski ConsidAutoLabProc21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d2\/Fig2_Liscouski_ConsidAutoLabProc21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Lab sample processing, initial data entry through results<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Most LIMS will permit sample submitters (with appropriate permissions) to enter the sample description information directly into the LIMS, reducing some of the clerical burden. Standardizing on sample containers, with barcodes, reduces the effort and cost in some aspects of sample handling. A barcode scanner could be used to scan samples as they arrive into the lab, letting the system know that they are ready to be tested.\n<\/p><p>That brings us to an evaluation of the process as a whole, as well as an examination of the individual steps in the procedure. As shown in Figure 1, automation can be done in one of two ways: automating the full process or automating individual steps. Your choice depends on several factors, not the least of which is your comfort level and confidence in adopting automation as a strategy for increasing productivity. For some, concentrating on improvements in individual steps is an attractive approach. The cost and risk may be lower and if a problem occurs you can always backup to a fully manual implementation until they are resolved.\n<\/p><p>Care does have to be taken in choosing which steps to improve. From one perspective, you\u2019d want to do the step-wise implementation of automation as close to the end of the process as possible. The problem with doing it earlier is that you may create a backup in later stages of the process. Optimizing step 2, for example, doesn\u2019t do you much good if step 3 is overloaded and requires more people, or additional (possibly unplanned) automation to relieve a bottleneck there. In short, before you automate or improve a given step, you need to be sure that downstream processing can absorb the increase in materials flow. In addition, optimizing all the individual steps, one at time, doesn\u2019t necessarily add up to a well-designed full system automation. The transition between steps may not be as effective or efficient if the system were evaluated as a whole. If the end of the process is carried out by commercial instrumentation, the ability to absorb more work is easier since most of these systems are automated with computer data acquisition and processing, and many have auto-samplers available to accumulate samples that can be processed automatically. Some of those auto-samplers have built in robotics for common sample handling functions. If the workload builds, additional instruments can pick up the load, and equipment such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Baytek_International,_Inc.\" title=\"Baytek International, Inc.\" class=\"wiki-link\" data-key=\"92bd2781da39f29dfafa73d5f07fd530\">Baytek International\u2019s<\/a> TurboTube<sup id=\"rdp-ebb-cite_ref-BaytekiPRO_22-0\" class=\"reference\"><a href=\"#cite_note-BaytekiPRO-22\">[15]<\/a><\/sup> can accumulate sample vials in a common system and route them to individual instruments for processing.\n<\/p><p>Another consideration for partial automation is where the process is headed in the future. If the need for the process persists over a long period of time, will you eventually get to the point of needing to redo the automation to an integrated stream? If so, is it better to take the plunge early on instead of continually expending resources to upgrade it?\n<\/p><p>Other considerations include the ability to re-purpose equipment. If a process isn\u2019t used full-time (a justification for partial automation) the same components may be used in improving other processes. Ideally, if you go the full-process automation route, you\u2019ll have sufficient sample throughput to keep it running for an extended period of time, and not have to start and stop the system as samples accumulate. A smoothly running slower automation process is better than a faster system that lies idle for significant periods of time, particularly since startup and shutdown procedures may diminish the operational cost savings in both equipment use and people\u2019s time.\n<\/p><p>All these points become part of both the technical justification and budget requirements.\n<\/p><p><b>Analyzing the process: Simulation and modeling<\/b>\n<\/p><p>Simulation and modeling have been part of science and engineering for decades, supported by ever-increasing powerful computing hardware and software. Continuous systems simulations have shown us the details of how machinery works, how chemical reactions occur, and how chromatographic systems and other instrumentation behaves.<sup id=\"rdp-ebb-cite_ref-JoyceComputer18_23-0\" class=\"reference\"><a href=\"#cite_note-JoyceComputer18-23\">[16]<\/a><\/sup> There is another aspect to modeling and simulation that is appropriate here.\n<\/p><p>Discrete-events simulation (DES) is used to model and understand processes in business and manufacturing applications, evaluating the interactions between service providers and customers, for example. One application of DES is to determine the best way to distribute incoming customers to a limited number of servers, taking into account that not all customers have the same needs; some will tie up a service provider a lot longer than others, as represented by the classic bank teller line problem. That is one question that discrete systems can analyze. This form of simulation and modeling is appropriate to event-driven processes where the action is focused on discrete steps (like materials moving from one workstation to another) rather than as a continuous function of time (most naturally occurring systems fall into this category, e.g., heat flow and models using differential equations).\n<\/p><p>The processes in your lab can be described and analyzed via DES systems.<sup id=\"rdp-ebb-cite_ref-CostigliolaSimul17_24-0\" class=\"reference\"><a href=\"#cite_note-CostigliolaSimul17-24\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MengImprov13_25-0\" class=\"reference\"><a href=\"#cite_note-MengImprov13-25\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JunApplic99_26-0\" class=\"reference\"><a href=\"#cite_note-JunApplic99-26\">[19]<\/a><\/sup> Those laboratory procedures are a sequence of steps, each having a precursor, variable duration, and following step until the end of the process is reached; this is basically the same as a manufacturing operation where modeling and simulation have been used successfully for decades. DES can be used to evaluate those processes and ask questions that can guide you on the best paths to take in applying automation technologies and solving productivity or throughput problems. For example:\n<\/p>\n<ul><li> What happens if we tighten up the variability in a particular step; how will that affect the rest of the system?<\/li>\n<li> What happens at the extremes of the variability in process steps; does it create a situation where samples pile up?<\/li>\n<li> How much of a workload can the process handle before one step becomes saturated with work and the entire system backs up?<\/li>\n<li> Can you introduce an alternate path to process those samples and avoid problems (e.g., if samples are held for too long in one stage, do they deteriorate)?<\/li>\n<li> Can the output of several parallel slower procedures be merged into a feed stream for a common instrumental technique?<\/li><\/ul>\n<p>In complex procedures some steps may be sensitive to small delays, and DES can help test and uncover them. Note that setting up these models will require the collection of a lot of data about the processes and their timing, so this is not something to be taken casually.\n<\/p><p>Previous research<sup id=\"rdp-ebb-cite_ref-JoyceComputer18_23-1\" class=\"reference\"><a href=\"#cite_note-JoyceComputer18-23\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CostigliolaSimul17_24-1\" class=\"reference\"><a href=\"#cite_note-CostigliolaSimul17-24\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MengImprov13_25-1\" class=\"reference\"><a href=\"#cite_note-MengImprov13-25\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JunApplic99_26-1\" class=\"reference\"><a href=\"#cite_note-JunApplic99-26\">[19]<\/a><\/sup> suggests only a few ideas where simulation can be effective, including one where an entire labs operation\u2019s was evaluated. Models that extensive can be used to not only look at procedures, but also the introduction of informatics systems. This may appear to be a significant undertaking, and it can be depending on the complexity of the lab processes. However, simple processes can be initially modeled on spreadsheets to see if more significant effort is justified. Operations research, of which DES is a part, has been usefully applied in production operations to increase throughput and improve ROI. It might be successfully applied to some routine production oriented lab work.\n<\/p><p>Most lab processes are linear in their execution, one step following another, with the potential for loop-backs should problems be recognized with samples, reagents (e.g., being out-of-date, doesn\u2019t look right, need to obtain new materials), or equipment (e.g., not functioning properly, out of calibration, busy due to other work). On one level, the modeling of a manually implemented process should appear to be simple: each step takes a certain amount of time. If you add up the times, you have a picture of the process execution through time. However, the reality is quite different if you take into account problems (and their resolution) that can occur in each of those steps. The data collection used to model the procedure can change how that picture looks and your ability to improve it. By monitoring the process over a number of iterations, you can find out how much variation there is in the execution time for each step and whether or not the variation is a normal distribution or skewed (e.g., if one step is skewed, how does it impact others?).\n<\/p><p>Questions to ask about potential problems that could occur at each step include:\n<\/p>\n<ul><li> How often do problems with reagents occur and how much of a delay does that create?<\/li>\n<li> Is instrumentation always in calibration (do you know?), are there operational problems with devices and their control systems (what are the ramifications?), are procedures delayed due to equipment being in use by someone else, and how long does it take to make changeovers in operating conditions?<\/li>\n<li> What happens to the samples; do they degrade over time? What impact does this have on the accuracy of results and their reproducibility?<\/li>\n<li> How often are workflows interrupted by the need to deal with high-priority samples, and what effect does it have on the processing of other samples?<\/li><\/ul>\n<p>Just the collection of data can suggest useful improvements before there are any considerations for automation, and perhaps negating the need for it. The answer to a lab\u2019s productivity might be as simple as adding another instrument if that is a bottleneck. It might also suggest that an underutilized device might be more productive if sample preparation for different procedures workflows were organized differently. Underutilization might be a consequence of the amount of time needed to prepare the equipment for service: doing so for one sample might be disproportionately time consuming (and expensive) and cause other samples to wait until there were enough of them to justify the preparation. It could also suggest that some lab processes should be outsourced to groups that have a more consistent sample flow and turn-around time (TAT) for that technique. Some of these points are illustrated in Figures 3a and 3b below.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3a_Liscouski_ConsidAutoLabProc21.png\" class=\"image wiki-link\" data-key=\"b54669998fa4961ccecbfd829e32d89d\"><img alt=\"Fig3a Liscouski ConsidAutoLabProc21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/af\/Fig3a_Liscouski_ConsidAutoLabProc21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3a.<\/b> Simplified process views versus some modeling considerations. Note that the total procedure execution time is affected by the variability in each step, plus equipment and material availability delays; these can change from one day to the next in manual implementations.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3b_Liscouski_ConsidAutoLabProc21.png\" class=\"image wiki-link\" data-key=\"dd6e3ce511391ee3f325577fdb4f9ca8\"><img alt=\"Fig3b Liscouski ConsidAutoLabProc21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/eb\/Fig3b_Liscouski_ConsidAutoLabProc21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3b.<\/b> The execution times of each step include the variable execution times of potential issues that can occur in each stage. Note that because each factor has a different distribution curve, the total execution time has a much wider variability than the individual factors.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>How does the simulation system work? Once you have all the data set up, the simulation runs thousands of times using random number generators to pick out variables in execution times for each component in each step. For example, if there is a one-in-ten chance a piece of equipment will be in use when needed, 10% of the runs will show that with each one picking a delay time based on the input delay distribution function. With a large number of runs, you can see where delays exist and how they impact the overall processes behavior. You can also adjust the factors (what happens if equipment delays are cut in half) and see the effect of doing that. By testing the system, you can make better judgments on how to apply your resources.\n<\/p><p>Some of the issues that surface may be things that lab personnel know about and just deal with. It isn\u2019t until the problems are looked at that the impact on operations are fully realized and addressed. Modeling and simulation may appear to be overkill for lab process automation, something reserved for large- scale production projects. The physical size of the project is not the key factor, it is the complexity of the system that matters and the potential for optimization.\n<\/p><p>One benefit of a well-structured simulation of lab processes is that it would provide a solid basis for making recommendations for project approval and budgeting. The most significant element in modeling and simulation is the initial data collection, asking lab personnel to record the time it takes to carry out steps. This isn\u2019t likely to be popular if they don\u2019t understand why it is being done and what the benefits will be to them and the lab; accurate information is essential. This is another case where \u201cbad data is worse than no data.\u201d\n<\/p><p><b>Guidleines for process automation<\/b>\n<\/p><p>There are two types of guidelines that will be of interest to those conducting automation work: those that help you figure out what to do and how to do it, and those that must be met to satisfy regulatory requirements (both those evaluated by internal or external groups or organizations).\n<\/p><p>The first is going to depend on the nature of the science and automation being done to support it. Equipment vendor community support groups can be of assistance. Additionally, professional groups like the Pharmaceutical Research and Manufacturers of America (PhRMA), International Society for Pharmaceutical Engineering (ISPE), and Parenteral Drug Association (PDA) in the pharmaceutical and biotechnology industrues, with similar organizations in other industries and other countries. This may seem like a large jump from laboratory work, but it is appropriate when we consider the ramification of full-process automation. You are essentially developing a manufacturing operation on a lab bench, and the same concerns that large-scale production have also apply here; you have to ensure that the process is maintained and in control. The same is true of manual or semi-automated lab work, but it is more critical in fully-automated systems because of the potential high volume of results that can be produced.\n<\/p><p>The second set is going to consist of regulatory guidelines from groups appropriate to your industry: the <a href=\"https:\/\/www.limswiki.org\/index.php\/Food_and_Drug_Administration\" title=\"Food and Drug Administration\" class=\"wiki-link\" data-key=\"e2be8927071ac419c0929f7aa1ede7fe\">Food and Drug Administration<\/a> (FDA), <a href=\"https:\/\/www.limswiki.org\/index.php\/United_States_Environmental_Protection_Agency\" title=\"United States Environmental Protection Agency\" class=\"wiki-link\" data-key=\"877b052e12328aa52f6f7c3f2d56f99a\">Environmental Protection Agency<\/a> (EPA), and <a href=\"https:\/\/www.limswiki.org\/index.php\/International_Organization_for_Standardization\" title=\"International Organization for Standardization\" class=\"wiki-link\" data-key=\"116defc5d89c8a55f5b7c1be0790b442\">International Organization for Standardization<\/a> (ISO), as well as international groups (e.g., <a href=\"https:\/\/www.limswiki.org\/index.php\/Good_Automated_Manufacturing_Practice\" title=\"Good Automated Manufacturing Practice\" class=\"wiki-link\" data-key=\"a0f3d9c5bc4e330dbaec137fbe7f5dbb\">GAMP<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Good_Automated_Laboratory_Practices\" title=\"Good Automated Laboratory Practices\" class=\"wiki-link\" data-key=\"bef4ea1fa3e792ccf7471f9f09b804e6\">GALP<\/a>) etc. The interesting point is that we are looking at a potentially complete automation scheme for a procedure; does that come under manufacturing or laboratory? The likelihood is that laboratory guidelines will apply since the work is being done within the lab's footprint; however, there are things that can be learned from their manufacturing counterparts that may assist in project management and documentation. One interesting consideration is what happens when fully automated testing, such as on-line analyzers, becomes integrated with both the lab and production or process control data\/information streams. Which regulatory guidelines apply? It may come down to who is responsible for managing and supporting those systems.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Scheduling_automation_projects\">Scheduling automation projects<\/span><\/h4>\n<p>There are two parts to the schedule issue: how long is it going to take to compete the project (dependent on the process and people), and when do you start? The second point will be addressed here.\n<\/p><p>The timing of an automated process coming online is important. If it comes on too soon, there may not be enough work to justify it\u2019s use, and startup\/shutdown procedures may create more work than the system saves. If it comes too late, people will be frustrated with a heavy workload while the system that was supposed to provide relief is under development. \n<\/p><p>In Figure 4, the blue line represents the growing need for sample\/material processing using a given laboratory procedure. Ideally, you\u2019d like the automated version to be available when that blue line crosses the \u201cautomation needed on-line\u201d level of processing requirements; this the point where the current (manual?) implementation can no longer meet the demands of sample throughput requirements.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Liscouski_ConsidAutoLabProc21.png\" class=\"image wiki-link\" data-key=\"a92c905b5cb118443f1fe9176f6b38f5\"><img alt=\"Fig4 Liscouski ConsidAutoLabProc21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a8\/Fig4_Liscouski_ConsidAutoLabProc21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Timing the development of an automated system<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Those throughput limits are something you are going to have to evaluate and measure on a regular basis and use to make adjustments to the planning process (accelerating or slowing it as appropriate). How fast is the demand growing and at what point will your current methods be overwhelmed? Hiring more people is one option, but then the lab's operating expenses increase due to the cost of people, equipment, and lab space.\n<\/p><p>Once we have an idea of when something has to be working, we can begin the process of planning; note: the planning can begin at any point, it would be good to get the preliminaries done as soon as a manual process is finalized so that you have an idea of what you\u2019ll be getting into. Those preliminaries include looking at equipment that might be used (keeping track of its development), training requirements, developer resources, and implementation strategies, all of which would be updated as new information becomes available. The \u201cwe\u2019ll-get-to-it-when-we-need-it\u201d approach is just going to create a lot of stress and frustration.\n<\/p><p>You need to put together a first-pass project plan so that you can detail what you know, and more importantly what you don\u2019t know. The goal is to have enough information, updated as noted above, so that you can determine if an automated solution is feasible, make an informed initial choice between full and partial automation, and have a timeline for implementation. Any time estimate is going to be subject to change as you gather information and refine your implementation approach. The point of the timeline is to figure out how long the yellow box in Figure 4 is because that is going to tell you how much time you have to get the plan together and working; it is a matter of setting priorities and recognizing what they are. The time between now and the start of the yellow box is what you have to work with for planning and evaluating plans, and any decisions that are needed before you begin, including corporate project management requirements and approvals.\n<\/p><p>Those plans have to include time for validation and the evaluation of the new implementation against the standard implementation. Does it work? Do we know how to use and maintain it? And are people educated in its use? Is there documentation for the project?\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Budgeting\">Budgeting<\/span><\/h4>\n<p>At some point, all the material above and following this section comes down to budgeting: how much will it cost to implement a program and is it worth it? Of the two points, the latter is the one that is most important. How do you go about that? (Note: Some of this material is also covered in the webinar series <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:A_Guide_for_Management:_Successfully_Applying_Laboratory_Systems_to_Your_Organization%27s_Work\" title=\"LII:A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work\" class=\"wiki-link\" data-key=\"00b300565027cb0518bcb0410d6df360\">A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work<\/a><\/i> in the section on ROI.)\n<\/p><p>What a lot of this comes down to is explaining and justifying the choices you\u2019ve made in your project proposal. We\u2019re not going to go into a lot of depth, but just note some of the key issues:\n<\/p>\n<ul><li> Did you choose full or partial automation for your process?<\/li>\n<li> What drove that choice? If in your view it would be less expensive than the full automation of a process, how long will it be until the next upgrade is needed to another stage?<\/li>\n<li> How independent are the potential, sequential implementation efforts that may be undertaken in the future? Will there be a need to connect them, and if so, how will the incremental costs compare to just doing it once and getting it over with?<\/li><\/ul>\n<p>There is a tendency in lab work to treat problems and the products that might be used to address them in isolation. You see the need for a LIMS or ELN, or an instrument data system, and the focus is on those issues. Effective decisions have to consider both the immediate and longer-term aspects of a problem. If you want to get access to a LIMS, have you considered how it will affect other aspects of lab work such as connecting instrument to it?\n<\/p><p>The same holds true for partial automation as a solution to a lab process productivity problem. While you are addressing a particular step, should you be looking at the potential for synergism by addressing other concerns. Modeling and simulations of processes can help resolve that issue.\n<\/p><p>Have you factored in the cost of support and education? The support issue needs to address the needs of lab personnel in managing the equipment and the options for vendor support, as well as the impact on IT groups. Note that the IT group will require access to vendor support, as well as being educated on their role in any project work.\n<\/p><p>What happens if you don\u2019t automate? One way to justify the cost of a project is to help people understand what the lab\u2019s operations will be like without it. Will more people, equipment, space, or added shifts be needed? At what cost? What would the impact be on those who need the results and how would it affect their programs?\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Build.2C_buy.2C_or_cooperate.3F\">Build, buy, or cooperate?<\/span><\/h2>\n<p>In this write up and some of the referenced materials, we\u2019ve noted several times the benefits that clinical labs have gained through automation, although crediting it all to the use of automation alone isn\u2019t fair. What the clinical laboratory industry did was recognize that there was a need for the use of automation to solve problems with the operational costs of running labs, and recognition that they could benefit further by coming together and cooperatively addressing lab operational problems.\n<\/p><p>It\u2019s that latter point that made the difference and resulted in standardized communications, and purpose-built commercial equipment that could be used to implement automation in their labs. They also had common sample types, common procedures, and data processing. That same commonality applies to segments of industrial and academic lab work. Take life sciences as an example. Where possible, that industry has standardized on micro-plates for sample processing. The result is a wide selection of instruments and robotics built around that sample-holding format that greatly improves lab economics and throughput. While it isn\u2019t the answer to everything, it\u2019s a good answer to a lot of things.\n<\/p><p>If your industry segment came together and recognized that you used common procedures, how would you benefit by creating a common approach to automation instead of each lab doing it on their own? It would open the development of common products or product variations from vendors and relieve the need for each lab developing its own answer to the need. The result could be more effective and easily supportable solutions.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Project_planning\">Project planning<\/span><\/h2>\n<p>Once you\u2019ve decided on the project you are going to undertake, the next stage is looking at the steps needed to manage your project (Figure 5).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Liscouski_ConsidAutoLabProc21.png\" class=\"image wiki-link\" data-key=\"63b05e9703977e362222fe94d673287c\"><img alt=\"Fig5 Liscouski ConsidAutoLabProc21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/4d\/Fig5_Liscouski_ConsidAutoLabProc21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Steps in a laboratory automation project. This diagram is modeled after the GAMP V for systems validation.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The planning begins with the method description from Figure 1, which describes the science behind the project and the specification of how the automation is expected to be put into effect: as full-process automation, a specific step, or steps in the process. The provider of those documents is considered the \u201ccustomer\u201d and is consistent with GAMP V nomenclature (Figure 6); that consistency is important due to the need for system-wide validation protocols.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Liscouski_ConsidAutoLabProc21.png\" class=\"image wiki-link\" data-key=\"71f06c8346c550b519af5eb78dccb441\"><img alt=\"Fig6 Liscouski ConsidAutoLabProc21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a2\/Fig6_Liscouski_ConsidAutoLabProc21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> GAMP V model for showing customer and supplier roles in specifying and evaluating project components for computer hardware and software.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>From there the \u201csupplier\u201d (e.g., internal development group, consultant, IT services, etc.) responds with a functional specification that is reviewed by the customer. The \u201canalysis, prototyping, and evaluation\u201d step, represented in the third box of Figure 5, is not the same as the process analysis noted earlier in this piece. The earlier section was to help you determine what work needed to be done and documented in the user requirements specification. The analysis and associated tasks here are specific to the implementation of this project. The colored arrows refer to the diagram in Figure 7. That process defines the equipment needed, dependencies, and options\/technologies for automation implementations, including robotics, instrument design requirements, pre-built automation (e.g., titrators, etc.) and any custom components. The documentation and specifications are part of the validation protocol.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Liscouski_ConsidAutoLabProc21.png\" class=\"image wiki-link\" data-key=\"71f06c8346c550b519af5eb78dccb441\"><img alt=\"Fig6 Liscouski ConsidAutoLabProc21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a2\/Fig6_Liscouski_ConsidAutoLabProc21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 7.<\/b> Defining dependencies and qualification of equipment<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The prototyping function is an important part of the overall process. It is rare that someone will look at a project and come up with a working solution on the first pass. There is always tinkering and modifications that occur as you move from a blank slate to a working system. You make notes along the way about what should be done differently in the final product, and places where improvements or adjustments are needed. These all become part of the input to the system design specification that will be reviewed and approved by the customer and supplier. The prototype can be considered a proof of concept or a demonstration of what will occur in the finished product. Remember also that prototypes would not have to be validated since they wouldn\u2019t be used in a production environment; they are simply a test bed used prior to the development of a production system.\n<\/p><p>The component design specifications are the refined requirement for elements that will be used in the final design. Those refinements could point to updated models of components or equipment used, modifications needed, or recommendations for products with capabilities other than those used in the prototype.\n<\/p><p>The boxes on the left side of Figure 5 are documents that go into increasing depth as the system is designed and specified. The details in those items will vary with the extent of the project. The right side of the diagram is a series of increasingly sophisticated testing and evaluation against steps in the right side, culminating in the final demonstration that the system works, has been validated, and is accepted by the customer. It also means that lab and support personnel are educated in their roles.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions_.28so_far.29\">Conclusions (so far)<\/span><\/h2>\n<p>\u201cLaboratory automation\u201d has to give way to \u201claboratory automation engineering.\u201d From the initial need to the completion of the validation process, we have to plan, design, and implement successful systems on a routine basis. Just as the manufacturing industries transitioned from cottage industries to production lines and then to integrated production-information systems, the execution of laboratory science has to tread a similar path if the demands for laboratory results are going to be met in a financially responsible manner. The science is fundamental; however, we need to pay attention now to efficient execution.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations.2C_acronyms.2C_and_initialisms\">Abbreviations, acronyms, and initialisms<\/span><\/h2>\n<p><b>AI<\/b>: Artificial intelligence\n<\/p><p><b>AuI<\/b>: Augmented intelligence\n<\/p><p><b>DES<\/b>: Discrete-events simulation\n<\/p><p><b>ELN<\/b>: Electronic laboratory notebook\n<\/p><p><b>EPA<\/b>: Environmental Protection Agency\n<\/p><p><b>FDA<\/b>: Food and Drug Administration\n<\/p><p><b>FRB<\/b>: Fast radio bursts\n<\/p><p><b>GALP<\/b>: Good automated laboratory practices\n<\/p><p><b>GAMP<\/b>: Good automated manufacturing practice\n<\/p><p><b>ISO<\/b>: International Organization for Standardization\n<\/p><p><b>LES<\/b>: Laboratory execution system\n<\/p><p><b>LIMS<\/b>: Laboratory information management system\n<\/p><p><b>ML<\/b>: Machine learning\n<\/p><p><b>ROI<\/b>: Return on investment\n<\/p><p><b>SDMS<\/b>: Scientific data management system\n<\/p><p><b>TAT<\/b>: Turn-around time\n<\/p><p><br \/>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Footnotes\">Footnotes<\/span><\/h2>\n<div class=\"reflist\" style=\"list-style-type: lower-alpha;\">\n<ol class=\"references\">\n<li id=\"cite_note-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-1\">\u2191<\/a><\/span> <span class=\"reference-text\">The term \"scientific manufacturing\" was first mentioned to the author by Mr. Alberto Correia, then of Cambridge Biomedical, Boston, MA.<\/span>\n<\/li>\n<li id=\"cite_note-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-8\">\u2191<\/a><\/span> <span class=\"reference-text\">Intelligent enterprise technologies referenced in the report include robotic process automation, machine learning, artificial intelligence, the internet Of things, predictive analysis, and cognitive computing.<\/span>\n<\/li>\n<li id=\"cite_note-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-10\">\u2191<\/a><\/span> <span class=\"reference-text\"><a rel=\"nofollow\" class=\"external text wiki-link\" href=\"https:\/\/en.wikipedia.org\/wiki\/Douglas_Engelbart\" data-key=\"91bc4cc7c8f10296b73c8689f9f470bb\">Doug Engelbart<\/a> found the field of human-computer interaction and is credited with the invention of the computer mouse, and the \u201cMother of All Demos\u201d in 1968.<\/span>\n<\/li>\n<li id=\"cite_note-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-18\">\u2191<\/a><\/span> <span class=\"reference-text\">See <a href=\"https:\/\/en.wikipedia.org\/wiki\/Metropolis_(1927_film)\" class=\"extiw wiki-link\" title=\"wikipedia:Metropolis (1927 film)\" data-key=\"0e4d0869e79f689fae15419230e0e902\"><i>Metropolis<\/i> (1927 film)<\/a> on Wikipedia.<\/span>\n<\/li>\n<li id=\"cite_note-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-19\">\u2191<\/a><\/span> <span class=\"reference-text\">See for example <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.projectmanager.com\/project-planning\" target=\"_blank\">https:\/\/www.projectmanager.com\/project-planning<\/a>; the simplest thing to do it put \u201cproject planning\u201d in a search engine and browse the results for something interesting.<\/span>\n<\/li>\n<li id=\"cite_note-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-20\">\u2191<\/a><\/span> <span class=\"reference-text\">See for example <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/theinformationdrivenlaboratory.wordpress.com\/category\/resources\/\" target=\"_blank\">https:\/\/theinformationdrivenlaboratory.wordpress.com\/category\/resources\/<\/a>; note that any references to the ILA should be ignored as the original site is gone, with the domain name perhaps having been leased by another organization that has no affiliation with the original Institute for Laboratory Automation.<\/span>\n<\/li>\n<li id=\"cite_note-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-21\">\u2191<\/a><\/span> <span class=\"reference-text\">As a starting point, view the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.artel.co\/\" target=\"_blank\">Artel, Inc. site<\/a> as one source. Also, John Bradshaw gave an <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.artel.co\/learning_center\/2589\/\" target=\"_blank\">informative presentation<\/a> on \u201cThe Importance of Liquid Handling Details and Their Impact on your Assays\u201d at the 2012 European Lab Automation Conference, Hamburg, Germany.<\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"About_the_author\">About the author<\/span><\/h2>\n<p>Initially educated as a chemist, author Joe Liscouski (joe dot liscouski at gmail dot com) is an experienced laboratory automation\/computing professional with over forty years of experience in the field, including the design and development of automation systems (both custom and commercial systems), LIMS, robotics and data interchange standards. He also consults on the use of computing in laboratory work. He has held symposia on validation and presented technical material and short courses on laboratory automation and computing in the U.S., Europe, and Japan. He has worked\/consulted in pharmaceutical, biotech, polymer, medical, and government laboratories. His current work centers on working with companies to establish planning programs for lab systems, developing effective support groups, and helping people with the application of automation and information technologies in research and quality control environments.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-FreyTheFuture13-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FreyTheFuture13_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Frey, C.B.; Osborne, M.A. (17 September 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.oxfordmartin.ox.ac.uk\/downloads\/academic\/The_Future_of_Employment.pdf\" target=\"_blank\">\"The Future of Employment: How Susceptible Are Jobs to Computerisation?\"<\/a> (PDF). Oxford Martin School, University of Oxford<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.oxfordmartin.ox.ac.uk\/downloads\/academic\/The_Future_of_Employment.pdf\" target=\"_blank\">https:\/\/www.oxfordmartin.ox.ac.uk\/downloads\/academic\/The_Future_of_Employment.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Future+of+Employment%3A+How+Susceptible+Are+Jobs+to+Computerisation%3F&rft.atitle=&rft.aulast=Frey%2C+C.B.%3B+Osborne%2C+M.A.&rft.au=Frey%2C+C.B.%3B+Osborne%2C+M.A.&rft.date=17+September+2013&rft.pub=Oxford+Martin+School%2C+University+of+Oxford&rft_id=https%3A%2F%2Fwww.oxfordmartin.ox.ac.uk%2Fdownloads%2Facademic%2FThe_Future_of_Employment.pdf&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HsuIsIt18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HsuIsIt18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Hsu, J. (24 September 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nbcnews.com\/mach\/science\/it-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586\" target=\"_blank\">\"Is it aliens? Scientists detect more mysterious radio signals from distant galaxy\"<\/a>. <i>NBC News MACH<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nbcnews.com\/mach\/science\/it-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586\" target=\"_blank\">https:\/\/www.nbcnews.com\/mach\/science\/it-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Is+it+aliens%3F+Scientists+detect+more+mysterious+radio+signals+from+distant+galaxy&rft.atitle=NBC+News+MACH&rft.aulast=Hsu%2C+J.&rft.au=Hsu%2C+J.&rft.date=24+September+2018&rft_id=https%3A%2F%2Fwww.nbcnews.com%2Fmach%2Fscience%2Fit-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TimmerAIPlus18-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TimmerAIPlus18_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Timmer, J. (18 July 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/arstechnica.com\/science\/2018\/07\/ai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work\/5\/\" target=\"_blank\">\"AI plus a chemistry robot finds all the reactions that will work\"<\/a>. <i>Ars Technica<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/arstechnica.com\/science\/2018\/07\/ai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work\/5\/\" target=\"_blank\">https:\/\/arstechnica.com\/science\/2018\/07\/ai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work\/5\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=AI+plus+a+chemistry+robot+finds+all+the+reactions+that+will+work&rft.atitle=Ars+Technica&rft.aulast=Timmer%2C+J.&rft.au=Timmer%2C+J.&rft.date=18+July+2018&rft_id=https%3A%2F%2Farstechnica.com%2Fscience%2F2018%2F07%2Fai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work%2F5%2F&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HelixAIHome-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HelixAIHome_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.askhelix.io\/\" target=\"_blank\">\"HelixAI - Voice Powered Digital Laboratory Assistants for Scientific Laboratories\"<\/a>. HelixAI<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.askhelix.io\/\" target=\"_blank\">http:\/\/www.askhelix.io\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=HelixAI+-+Voice+Powered+Digital+Laboratory+Assistants+for+Scientific+Laboratories&rft.atitle=&rft.pub=HelixAI&rft_id=http%3A%2F%2Fwww.askhelix.io%2F&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PharmaIQNewsAutom18-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PharmaIQNewsAutom18_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">PharmaIQ News (20 August 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.pharma-iq.com\/pre-clinical-discovery-and-development\/news\/automation-iot-and-the-future-of-smarter-research-environments\" target=\"_blank\">\"Automation, IoT and the future of smarter research environments\"<\/a>. <i>PharmaIQ<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.pharma-iq.com\/pre-clinical-discovery-and-development\/news\/automation-iot-and-the-future-of-smarter-research-environments\" target=\"_blank\">https:\/\/www.pharma-iq.com\/pre-clinical-discovery-and-development\/news\/automation-iot-and-the-future-of-smarter-research-environments<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Automation%2C+IoT+and+the+future+of+smarter+research+environments&rft.atitle=PharmaIQ&rft.aulast=PharmaIQ+News&rft.au=PharmaIQ+News&rft.date=20+August+2018&rft_id=https%3A%2F%2Fwww.pharma-iq.com%2Fpre-clinical-discovery-and-development%2Fnews%2Fautomation-iot-and-the-future-of-smarter-research-environments&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PharmaIQTheFuture17-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PharmaIQTheFuture17_7-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-PharmaIQTheFuture17_7-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">PharmaIQ (14 November 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.pharma-iq.com\/pre-clinical-discovery-and-development\/whitepapers\/the-future-of-drug-discovery-ai-2020\" target=\"_blank\">\"The Future of Drug Discovery: AI 2020\"<\/a>. PharmaIQ<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.pharma-iq.com\/pre-clinical-discovery-and-development\/whitepapers\/the-future-of-drug-discovery-ai-2020\" target=\"_blank\">https:\/\/www.pharma-iq.com\/pre-clinical-discovery-and-development\/whitepapers\/the-future-of-drug-discovery-ai-2020<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Future+of+Drug+Discovery%3A+AI+2020&rft.atitle=&rft.aulast=PharmaIQ&rft.au=PharmaIQ&rft.date=14+November+2017&rft.pub=PharmaIQ&rft_id=https%3A%2F%2Fwww.pharma-iq.com%2Fpre-clinical-discovery-and-development%2Fwhitepapers%2Fthe-future-of-drug-discovery-ai-2020&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RossettoFight18-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RossettoFight18_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rossetto, L. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.magzter.com\/stories\/Science\/WIRED\/Fight-The-Dour\" target=\"_blank\">\"Fight the Dour\"<\/a>. <i>Wired<\/i> (October): 826\u20137<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.magzter.com\/stories\/Science\/WIRED\/Fight-The-Dour\" target=\"_blank\">https:\/\/www.magzter.com\/stories\/Science\/WIRED\/Fight-The-Dour<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fight+the+Dour&rft.jtitle=Wired&rft.aulast=Rossetto%2C+L.&rft.au=Rossetto%2C+L.&rft.date=2018&rft.issue=October&rft.pages=826%E2%80%937&rft_id=https%3A%2F%2Fwww.magzter.com%2Fstories%2FScience%2FWIRED%2FFight-The-Dour&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrukerOPUS-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrukerOPUS_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bruker.com\/en\/products-and-solutions\/infrared-and-raman\/opus-spectroscopy-software\/search-identify.html\" target=\"_blank\">\"OPUS Package: SEARCH & IDENT\"<\/a>. Bruker Corporation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bruker.com\/en\/products-and-solutions\/infrared-and-raman\/opus-spectroscopy-software\/search-identify.html\" target=\"_blank\">https:\/\/www.bruker.com\/en\/products-and-solutions\/infrared-and-raman\/opus-spectroscopy-software\/search-identify.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OPUS+Package%3A+SEARCH+%26+IDENT&rft.atitle=&rft.pub=Bruker+Corporation&rft_id=https%3A%2F%2Fwww.bruker.com%2Fen%2Fproducts-and-solutions%2Finfrared-and-raman%2Fopus-spectroscopy-software%2Fsearch-identify.html&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BourneMyBoss13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BourneMyBoss13_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bourne, D. (2013). \"My Boss the Robot\". <i>Scientific American<\/i> <b>308<\/b> (5): 38\u201341. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fscientificamerican0513-38\" target=\"_blank\">10.1038\/scientificamerican0513-38<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23627215\" target=\"_blank\">23627215<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=My+Boss+the+Robot&rft.jtitle=Scientific+American&rft.aulast=Bourne%2C+D.&rft.au=Bourne%2C+D.&rft.date=2013&rft.volume=308&rft.issue=5&rft.pages=38%E2%80%9341&rft_id=info:doi\/10.1038%2Fscientificamerican0513-38&rft_id=info:pmid\/23627215&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SalesForceSecondAnn17-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SalesForceSecondAnn17_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">SalesForce Research (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/a.sfdcstatic.com\/content\/dam\/www\/ocms\/assets\/pdf\/misc\/2017-state-of-it-report-salesforce.pdf\" target=\"_blank\">\"Second Annual State of IT\"<\/a> (PDF). SalesForce<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/a.sfdcstatic.com\/content\/dam\/www\/ocms\/assets\/pdf\/misc\/2017-state-of-it-report-salesforce.pdf\" target=\"_blank\">https:\/\/a.sfdcstatic.com\/content\/dam\/www\/ocms\/assets\/pdf\/misc\/2017-state-of-it-report-salesforce.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Second+Annual+State+of+IT&rft.atitle=&rft.aulast=SalesForce+Research&rft.au=SalesForce+Research&rft.date=2017&rft.pub=SalesForce&rft_id=https%3A%2F%2Fa.sfdcstatic.com%2Fcontent%2Fdam%2Fwww%2Focms%2Fassets%2Fpdf%2Fmisc%2F2017-state-of-it-report-salesforce.pdf&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SealAnal-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SealAnal_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/seal-analytical.com\/Products\/tabid\/55\/language\/en-US\/Default.aspx\" target=\"_blank\">\"Seal Analytical - Products\"<\/a>. Seal Analytical<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/seal-analytical.com\/Products\/tabid\/55\/language\/en-US\/Default.aspx\" target=\"_blank\">https:\/\/seal-analytical.com\/Products\/tabid\/55\/language\/en-US\/Default.aspx<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Seal+Analytical+-+Products&rft.atitle=&rft.pub=Seal+Analytical&rft_id=https%3A%2F%2Fseal-analytical.com%2FProducts%2Ftabid%2F55%2Flanguage%2Fen-US%2FDefault.aspx&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BranLuebbe-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BranLuebbe_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.spxflow.com\/bran-luebbe\/\" target=\"_blank\">\"Bran+Luebbe\"<\/a>. SPX FLOW, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.spxflow.com\/bran-luebbe\/\" target=\"_blank\">https:\/\/www.spxflow.com\/bran-luebbe\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bran%2BLuebbe&rft.atitle=&rft.pub=SPX+FLOW%2C+Inc&rft_id=https%3A%2F%2Fwww.spxflow.com%2Fbran-luebbe%2F&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DouglasScientificArrayTape-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DouglasScientificArrayTape_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.douglasscientific.com\/Products\/ArrayTape.aspx\" target=\"_blank\">\"Array Tape Advanced Consumable\"<\/a>. Douglas Scientific<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.douglasscientific.com\/Products\/ArrayTape.aspx\" target=\"_blank\">https:\/\/www.douglasscientific.com\/Products\/ArrayTape.aspx<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Array+Tape+Advanced+Consumable&rft.atitle=&rft.pub=Douglas+Scientific&rft_id=https%3A%2F%2Fwww.douglasscientific.com%2FProducts%2FArrayTape.aspx&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Agilent1200Series-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Agilent1200Series_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.agilent.com\/cs\/library\/usermanuals\/Public\/G1329-90012_StandPrepSamplers_ebook.pdf\" target=\"_blank\">\"Agilent 1200 Series Standard and Preparative Autosamplers - User Manual\"<\/a> (PDF). Agilent Technologies. November 2008<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.agilent.com\/cs\/library\/usermanuals\/Public\/G1329-90012_StandPrepSamplers_ebook.pdf\" target=\"_blank\">https:\/\/www.agilent.com\/cs\/library\/usermanuals\/Public\/G1329-90012_StandPrepSamplers_ebook.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Agilent+1200+Series+Standard+and+Preparative+Autosamplers+-+User+Manual&rft.atitle=&rft.date=November+2008&rft.pub=Agilent+Technologies&rft_id=https%3A%2F%2Fwww.agilent.com%2Fcs%2Flibrary%2Fusermanuals%2FPublic%2FG1329-90012_StandPrepSamplers_ebook.pdf&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaytekiPRO-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BaytekiPRO_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.baytekinternational.com\/products\/ipro-interface\/89-products\" target=\"_blank\">\"iPRO Interface - Products\"<\/a>. Baytek International, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.baytekinternational.com\/products\/ipro-interface\/89-products\" target=\"_blank\">https:\/\/www.baytekinternational.com\/products\/ipro-interface\/89-products<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=iPRO+Interface+-+Products&rft.atitle=&rft.pub=Baytek+International%2C+Inc&rft_id=https%3A%2F%2Fwww.baytekinternational.com%2Fproducts%2Fipro-interface%2F89-products&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JoyceComputer18-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JoyceComputer18_23-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-JoyceComputer18_23-1\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Joyce, J. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labmanager.com\/laboratory-technology\/computer-modeling-and-simulation-1826\" target=\"_blank\">\"Computer Modeling and Simulation\"<\/a>. <i>Lab Manager<\/i> (9): 32\u201335<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.labmanager.com\/laboratory-technology\/computer-modeling-and-simulation-1826\" target=\"_blank\">https:\/\/www.labmanager.com\/laboratory-technology\/computer-modeling-and-simulation-1826<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Computer+Modeling+and+Simulation&rft.jtitle=Lab+Manager&rft.aulast=Joyce%2C+J.&rft.au=Joyce%2C+J.&rft.date=2018&rft.issue=9&rft.pages=32%E2%80%9335&rft_id=https%3A%2F%2Fwww.labmanager.com%2Flaboratory-technology%2Fcomputer-modeling-and-simulation-1826&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CostigliolaSimul17-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CostigliolaSimul17_24-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-CostigliolaSimul17_24-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Costigliola, A.; Ata\u00edde, F.A.P.; Vieira, S.M. et al. (2017). \"Simulation Model of a Quality Control Laboratory in Pharmaceutical Industry\". <i>IFAC-PapersOnLine<\/i> <b>50<\/b> (1): 9014-9019. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ifacol.2017.08.1582\" target=\"_blank\">10.1016\/j.ifacol.2017.08.1582<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Simulation+Model+of+a+Quality+Control+Laboratory+in+Pharmaceutical+Industry&rft.jtitle=IFAC-PapersOnLine&rft.aulast=Costigliola%2C+A.%3B+Ata%C3%ADde%2C+F.A.P.%3B+Vieira%2C+S.M.+et+al.&rft.au=Costigliola%2C+A.%3B+Ata%C3%ADde%2C+F.A.P.%3B+Vieira%2C+S.M.+et+al.&rft.date=2017&rft.volume=50&rft.issue=1&rft.pages=9014-9019&rft_id=info:doi\/10.1016%2Fj.ifacol.2017.08.1582&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MengImprov13-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MengImprov13_25-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-MengImprov13_25-1\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Meng, L.; Liu, R.; Essick, C. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchgate.net\/publication\/263238201_Improving_Medical_Laboratory_Operations_via_Discrete-event_Simulation\" target=\"_blank\">\"Improving Medical Laboratory Operations via Discrete-event Simulation\"<\/a>. <i>Proceedings of the 2013 INFORMS Healthcare Conference<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.researchgate.net\/publication\/263238201_Improving_Medical_Laboratory_Operations_via_Discrete-event_Simulation\" target=\"_blank\">https:\/\/www.researchgate.net\/publication\/263238201_Improving_Medical_Laboratory_Operations_via_Discrete-event_Simulation<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improving+Medical+Laboratory+Operations+via+Discrete-event+Simulation&rft.jtitle=Proceedings+of+the+2013+INFORMS+Healthcare+Conference&rft.aulast=Meng%2C+L.%3B+Liu%2C+R.%3B+Essick%2C+C.+et+al.&rft.au=Meng%2C+L.%3B+Liu%2C+R.%3B+Essick%2C+C.+et+al.&rft.date=2013&rft_id=https%3A%2F%2Fwww.researchgate.net%2Fpublication%2F263238201_Improving_Medical_Laboratory_Operations_via_Discrete-event_Simulation&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JunApplic99-26\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JunApplic99_26-0\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-JunApplic99_26-1\">19.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">\"Application of discrete-event simulation in health care clinics: A survey\". <i>Journal of the Operational Research Society<\/i> <b>50<\/b>: 109\u201323. 1999. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1057%2Fpalgrave.jors.2600669\" target=\"_blank\">10.1057\/palgrave.jors.2600669<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Application+of+discrete-event+simulation+in+health+care+clinics%3A+A+survey&rft.jtitle=Journal+of+the+Operational+Research+Society&rft.date=1999&rft.volume=50&rft.pages=109%E2%80%9323&rft_id=info:doi\/10.1057%2Fpalgrave.jors.2600669&rfr_id=info:sid\/en.wikipedia.org:LII:Considerations_in_the_Automation_of_Laboratory_Procedures\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n\n<!-- \nNewPP limit report\nCached time: 20210429194129\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.517 seconds\nReal time usage: 2.589 seconds\nPreprocessor visited node count: 12096\/1000000\nPreprocessor generated node count: 20027\/1000000\nPost\u2010expand include size: 77974\/2097152 bytes\nTemplate argument size: 30704\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 266.695 1 - -total\n 88.66% 236.463 2 - Template:Reflist\n 71.18% 189.842 19 - Template:Citation\/core\n 48.34% 128.916 13 - Template:Cite_web\n 27.15% 72.417 6 - Template:Cite_journal\n 4.65% 12.401 7 - Template:Efn\n 4.12% 10.978 22 - Template:Citation\/make_link\n 2.82% 7.512 4 - Template:Citation\/identifier\n 0.97% 2.587 8 - Template:Hide_in_print\n 0.69% 1.840 4 - Template:Only_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12315-0!*!0!!en!5!* and timestamp 20210429194127 and revision id 41798\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/LII:Considerations_in_the_Automation_of_Laboratory_Procedures\">https:\/\/www.limswiki.org\/index.php\/LII:Considerations_in_the_Automation_of_Laboratory_Procedures<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","e0147011cc1eb892e1a35e821657a6d9_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/31\/Fig1_Liscouski_ConsidAutoLabProc21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d2\/Fig2_Liscouski_ConsidAutoLabProc21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/af\/Fig3a_Liscouski_ConsidAutoLabProc21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/eb\/Fig3b_Liscouski_ConsidAutoLabProc21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a8\/Fig4_Liscouski_ConsidAutoLabProc21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/4d\/Fig5_Liscouski_ConsidAutoLabProc21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a2\/Fig6_Liscouski_ConsidAutoLabProc21.png"],"e0147011cc1eb892e1a35e821657a6d9_timestamp":1619725287,"655f7d48a642e9b45533745af73f0d59_type":"article","655f7d48a642e9b45533745af73f0d59_title":"Laboratory technology planning and management: The practice of laboratory systems engineering (Liscouski 2020)","655f7d48a642e9b45533745af73f0d59_url":"https:\/\/www.limswiki.org\/index.php\/LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering","655f7d48a642e9b45533745af73f0d59_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tLII:Laboratory Technology Planning and Management: The Practice of Laboratory Systems Engineering\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tTitle: Laboratory Technology Planning and Management: The Practice of Laboratory Systems Engineering\nAuthor for citation: Joe Liscouski, with editorial modifications by Shawn Douglas\nLicense for content: Creative Commons Attribution 4.0 International\nPublication date: December 2020\n\nContents\n\n1 Introduction \n\n1.1 Directions in lab operations \n\n1.1.1 The lab of the future \n1.1.2 Trends in science applications \n\n\n1.2 Education \n\n\n2 Making laboratory informatics and automation work \n\n2.1 Introduction to this section \n\n2.1.1 The point of planning \n2.1.2 Who is responsible for laboratory technology planning and management (TPM)? \n2.1.3 Why put so much effort into planning and technology management? \n\n\n\n\n3 Different ways of looking at laboratories \n4 Labs in transition, from manual operation to modern facilities \n\n4.1 There's a plan for that? \n4.2 Thinking about a model for lab operations \n\n\n5 The seven goals of planning and managing lab technologies \n\n5.1 First goal: Support an environment that fosters productivity and innovation \n5.2 Second goal: Develop high-quality data and information \n5.3 Third goal: Manage K\/I\/D effectively, putting them in a structure that encourages use and protects value \n5.4 Fourth goal: Ensure a high level of data integrity at every step \n\n5.4.1 Definitions of data integrity \n\n\n5.5 Fifth goal: Addressing security throughout the lab \n\n5.5.1 Examine aspects of the facility itself \n5.5.2 Security and the working environment \n5.5.3 Summary \n\n\n5.6 Sixth goal: Acquiring and developing \"products\" that support regulatory requirements \n\n5.6.1 The importance of documentation \n5.6.2 Additional considerations in creating supportable systems and processes \n5.6.3 Product life cycles \n5.6.4 Summary \n\n\n5.7 Seventh goal: Addressing systems integration and harmonization \n\n5.7.1 Harmonization \n\n\n\n\n6 Laboratory systems engineers \n7 Closing \n8 Abbreviations, acronyms, and initialisms \n9 Footnotes \n10 About the author \n11 References \n\n\n\nIntroduction \nWhat separates successful advanced laboratories from all the others? It's largely their ability to meet their goals, with the effective use of resources: people, time, money, equipment, data, and information. The fundamental goals of laboratory work haven\u2019t changed, but they are under increased pressure to do more and do it faster, with a better return on investment (ROI). Laboratory managers have turned to electronic technologies (e.g., computers, networks, robotics, microprocessors, database systems, etc.) to meet those demands. However, without effective planning, technology management, and education, those technologies will only get labs part of the way to meeting their needs. We need to learn how to close the gap between getting part-way there and getting where we need to be. The practice of science has changed; we need to meet that change to be successful.\nThis document was written to get people thinking more seriously about the technologies used in laboratory work and how those technologies contribute to meeting the challenges labs are facing. There are three primary concerns:\n\n The need for planning and management: When digital components began to be added to lab systems, it was a slow incremental process: integrators and microprocessors grew in capability as the marketplace accepted them. That development gave us the equipment we have now, equipment that can be used in isolation or in a networked, integrated system. In either case, they need attention in their application and management to protect electronic laboratory data, ensure that it can be effectively used, and ensure that the systems and products put in place are both the right ones, and that they fully contribute to improvements in lab operations.\n The need for more laboratory systems engineers (LSEs): There is increasing demand for people who have the education and skills needed to accomplish the points above and provide research and testing groups with the support they need.[a]\n The need to collaborate with vendors: In order to develop the best products needed for laboratory work, vendors should be provided more user input. Too often vendors have an idea for a product or modifications to existing products, yet they lack a fully qualified audience to bounce ideas off of. With the planning in the first concern in place, we should be able to approach vendors and say, with confidence, \"this is what is needed\" and explain why.\nIf the audience for this work were product manufacturing or production facilities, everything that was being said would have been history. The efficiency and productivity of production operations directly impacts profitability and customer satisfaction; the effort to optimize operations would have been an essential goal. When it comes to laboratory operations, that same level of attention found in production operations must be in place to accelerate laboratory research and testing operations, reducing cost and improving productivity. Aside from a few lab installations in large organizations, this same level of attention isn\u2019t given, as people aren\u2019t educated as to its importance. The purpose of this work is to present ideas of what laboratory technology challenges can be addressed through planning activities using a series of goals.\nThis material is an expansion upon two presentations:\n\n \"Laboratory Technology Management & Planning,\" 2nd Annual Lab Asset & Facility Management in Pharma 2019, San Diego, CA, October 22, 2019\n \"How Digital Technologies are Changing the Landscape of Lab Operations,\" Lab Manager webinar, April 2020\nDirections in lab operations \nThe lab of the future \nPeople often ask what the lab of the future (LOF) is going to look like, as if there were a design or model that we should be aspiring toward. There isn\u2019t. Your lab's future is in your hands to mold, a blank sheet of paper upon which you define your lab's future by setting objectives, developing a functional physical and digital architecture, planning processes and implementations, and managing technology that supports both scientific and laboratory information management. If that sound scary, it\u2019s understandable. But you must take the time to educate yourself and bring in people (e.g., LSEs, consultants, etc.) who can assist you.\nToo often, if vendors and consultants are asked what the LOF is going to look like, the response lines up with their corporate interests. No one knows what the LOF is because there isn\u2019t a singular future, but rather different futures for different types of labs. (Just think of all the different scientific disciplines that exist; one future doesn\u2019t fit all.) Your lab's future is in your hands. What do you want it to be?\nThe material in this document isn\u2019t intended to define your LOF, but to help you realize it once the framework has been created, and you are in the best position to create it. As you create that framework, you'll be asking:\n\n Are you satisfied with your lab's operations? What works and what doesn\u2019t? What needs fixing and how shall it be prioritized?\n Has management raised any concerns?\n What do those working in the lab have to say?\n How is your lab going to change in the next one to five years?\n Does your industry have a working group for lab operations, computing, and automation?\nAdding to question five, many companies tend to keep the competition at arm's length, minimizing contact for fear of divulging confidential information. However, if practically everyone is using the same set of test procedures from a trusted neutral source (e.g., ASTM International, United States Pharmacopeia, etc.), there\u2019s nothing confidential there. Instead of developing automated versions of the same procedure independently, companies can join forces, spread the cost, and perhaps come up with a better solution. With that effort as a given, you collectively have something to approach the vendor community with and say \u201cwe need this modification or new product.\u201d This is particularly beneficial to the vendor when they receive a vetted product requirements document to work from.\nAgain, you don\u2019t wait for the lab of the future to happen, you create it. If you want to see the direction lab operations in the future can take, look to the manufacturing industry: it has everything from flexible manufacturing, cooperative robotics[1][2], and so on.[b] This is appropriate in both basic and applied research, as well as quality control.\nBoth manufacturing and lab work are process-driven with a common goal: a high-quality product whose quality can be defended through appeal to process and data integrity.\nLab work can be broadly divided into two activities, with parallels to manufacturing: experimental procedure development (akin to manufacturing process development) and procedure execution (product production). (Note: Administrative work is part of lab operations but not an immediate concern here.) As such, we have to address the fact that lab work is part original science and part production work based on that science, e.g., as seen with quality control, clinical chemistry, and high-throughput screening labs. The routine production work of these and other labs can benefit most from automation efforts. We need to think more broadly about the use of automation technologies\u2014driving their development\u2014instead of waiting to see what vendors develop. \nWhere manufacturing and lab work differ is in the scale of the work environment, the nature of the work station equipment, the skills needed to carry out the work, and the adaptability of those doing the work to unexpected situations.\nMy hope is that this guide will get laboratory managers and other stakeholders to begin thinking more about planning and technology management, as well as the need for more education in that work.\n\nTrends in science applications \nIf new science isn\u2019t being developed, vendors will add digital hardware and software technology to existing equipment to improve capabilities and ease-of-use, separating themselves from the competition. However, there is still an obvious need for an independent organization to evaluate that technology (i.e., the lab version of Consumer Reports); as is, that evaluation process, done properly, would be time consuming for individual labs and would require a consistent methodology. With the increased use of automation, we need to do this better, such that the results can be used more widely (rather than every lab doing their own thing) and with more flexibility, using specialized equipment designed for automation applications.\nArtificial intelligence (AI) and machine learning (ML) are two other trending topics, but they are not quite ready for widespread real-world applications. However, modern examples still exist:\n\n Having a system that can bring up all relevant information on a research question\u2014a sort of super Google\u2014or a variation of IBM\u2019s Watson could have significant benefits.\n Analyzing complex data or large volumes of data could be beneficial, e.g., the analysis of radio astronomy data to find fast radio bursts (FRB).[3]\n \"[A] team at Glasgow University has paired a machine-learning system with a robot that can run and analyze its own chemical reaction. The result is a system that can figure out every reaction that's possible from a given set of starting materials.\"[4]\n HelixAI is using Amazon's Alexa as a digital assitant for laboratory work.[5]\nHowever there are problems using these technologies. ML systems have been shown to be susceptible to biases in their output depending on the nature and quality of the training materials. As for AI, at least in the public domain, we really don\u2019t know what that is, and what we think it is keeps changing as purported example emerge. One large problem for lab use is whether or not you can trust the results of an AI's output. We are used to the idea that lab systems and methods have to be validated before they are trusted, so how do you validate a system based on ML or AI?\n\nEducation \nThe major issue in all of this is having people educated to the point where they can successfully handle the planning and management of laboratory technology. One key point: most lab management programs focus on personnel issues, but managers also have to understand the capabilities and limitations of information technology and automation systems.\nOne result of the COVID-19 pandemic is that we are seeing the limitations of the four-year undergraduate degree program in science and engineering, as well as the state of remote learning. With the addition of information technologies, general systems thinking and modeling[c], statistical experimental design, and statistical process control have become multidisciplinary fields. We need options for continuing education throughout people\u2019s careers so they can maintain their competence and learn new material as needed.\n\nMaking laboratory informatics and automation work \nMaking laboratory informatics and automation work? \"Isn\u2019t that a job for IT or lab personnel?\" someone might ask. One of the problems in modern science is the development of specialists in disciplines. The laboratory and IT fields have many specialties, and specialists can be very good within those areas while at the same time not having an appreciation of wider operational issues. Topics like lab operations, technology management, and planning aren\u2019t covered in formal education courses, and they're often not well-covered in short courses or online programs.\n\u201cMaking it work\u201d depends on planning performed at a high enough level in the organization to encompass all affected facilities and departments, including information technology (IT) and facilities management. This wider perspective gives us the potential for synergistic operations across labs, consistent policies for facilities management and IT, and more effective use of outside resources (e.g., lab information technology support staff [LAB-IT], laboratory automation engineers [LAEs][d], equipment vendors, etc.). \nWe need to apply the same diligence to planning lab operations as we do any other critical corporate resource. Planning provides a structure for enabling effective and successful lab operations.\n\nIntroduction to this section \nThe common view of science laboratories is that of rooms filled with glassware, lab benches, and instruments being used by scientists to carry out experiments. While this is a reasonable perspective, what isn\u2019t as visually obvious is the end result of that work: the development of knowledge, information, and data.\nThe progress of laboratory work\u2014as well as the planning, documentation, analytical results related to that work\u2014have been recorded in paper-based laboratory notebooks for generations, and people are still using them today. However, these aren't the only paper records that have existed and are still in use; scientists also depend on charts, log books, administrative records, reports, indexes, and reference material. The latter half of the twentieth century introduced electronics into the lab and with it electronic recording in the form of computers and data storage systems. Early adopters of these technologies had to extend their expertise into the information technology realm because there were few people who understood both these new devices and their application to lab work\u2014you had to be an expert in both laboratory science and computer science.\nIn the 1980s and 90s, computers became commonplace and where once you had to understand hardware, software, operating systems, programming and application packages, you then simply had to know how to turn them on; no more impressive arrays of blinking lights, just a blinking cursor waiting for you to do something.\nAs systems gained ease-of-use, however, we lost the basic understanding of what these systems were and what they did, that they had faults, and that if we didn\u2019t plan for their effective use and counter those faults, we were opening ourselves to unpleasant surprises. The consequences at times were system crashes, lost data, and a lack of a real understanding of how the output of an instrument was transformed into a set of numbers, which meant we couldn\u2019t completely account for the results we were reporting. \nWe need to step back, take control, and institute effective technology planning and management, with appropriate corresponding education, so that the various data we are putting into laboratory informatics technologies have the desired outcome. We need to ensure that these technologies are providing a foundation for improving laboratory operations efficiency and a solid return on investment (ROI), while substantively advancing your business' ability to work and be productive. That's the purpose of the work we'll be discussing.\n\nThe point of planning \nThe point of planning and technology management is pretty simple: to ensure ...\n\n that the right technologies are in people's hands when they need them, and\n that those technologies complement each other as much as possible.\nThese are straightforward statements with a lot packed into them.\nRegarding the first point, the key words are \u201cthe right technologies.\u201d In order to define what that means, lab personnel have to understand the technologies in question and how they apply to their work. If those personnel have used or were taught about the technologies under consideration, it should be easy enough to do. However, laboratory informatics doesn\u2019t fall into that basket of things. The level of understanding has to be more than superficial. While personnel don\u2019t have to be software developers, they do have to understand what is happening within informatics systems, and how data processing handles their data and produces results. Determining the \u201cright technologies\u201d depends on the quality and depth of education possessed by lab personnel, and eventually by lab information technology support staff (LAB-IT?) as they become involved in the selection process.\nThe second point also has a lot buried inside it. Lab managers and personnel are used to specifying and purchasing items (e.g., instruments) as discrete tools. When it comes to laboratory informatics, we\u2019re working with things that connect to each other, in addition to performing a task. When we explore those connections, we need to assess how they are made, what we expect to gain, what compatibility issues exist, how to support them, how to upgrade them, what their life cycle is, etc. Most of the inter-connected devices people encounter in their daily lives are things that were expected to be connected with using a limited set of choices; the vendors know what those choices are and make it easy to do so, or otherwise their products won\u2019t sell. The laboratory technology market, on the other hand, is too open-ended. The options for physical connections might be there, but are they the right ones, and will they work? Do you have a good relationship with your IT people, and are they able to help (not a given)? Again, education is a major factor.\n\nWho is responsible for laboratory technology planning and management (TPM)? \nWhen asking who is responsible for TPM, the question really is \"who are the TPM stakeholders,\" or \"who has an invested interest in seeing TPM prove successful?\"\n\n Corporate or organizational management: These stakeholders set priorities and authorize funding, while also rationalizing and coordinating goals between groups. Unless the organization has a strong scientific base, they may not appreciate the options and benefits of TPM in lab work, or the possibilities of connecting the lab into the rest of the corporate data structure.\n Laboratory management: These stakeholders are responsible for developing and implementing plans, as well as translating corporate goals into lab priorities.\n Laboratory personnel: These stakeholders are the ones that actually do the work. However, they are in the best position to understand where technologies can be applied. They would also be relied on to provide user requirements documents for new projects and meet both internal and external (e.g., Food and Drug Administration [FDA], Environmental Protection Agency [EPA], International Organization for Standardization [ISO], etc.) performance guidelines.\n IT management and their support staff: While these stakeholders' traditional role is the support of computers, connected devices (e.g., printers, etc.) and network infrastructure, they may also be the first line of support for computers connected to lab equipment. IT staff either need to be educated to meet that need and support lab personnel, or have additional resources available to them. They may also be asked to participate in planning activities as subject matter experts on computing hardware and software.\n LAB-IT specialists: These stakeholders act as the \"additional resources\" alluded to in the previous point. These are crossover specialists that span the lab and IT spaces and can provide informed support to both. In most organizations, aside from large science-based companies, this isn\u2019t a real \"position,\" although once stated, its role is immediately recognized. In the past, I\u2019ve also referenced these stakeholders as being \u201claboratory automation engineers.\u201d[6]\n Facility management: These stakeholders need to ensure that the facilities support the evolving state of laboratory workspace requirements as traditional formats change to support robotics, instrumentation, computers, material flow, power, and HVAC requirements.\nCarrying out this work is going to rely heavily on expanding the education of those participating in the planning work; the subject matter goes well beyond material covered in degree programs.\n\nWhy put so much effort into planning and technology management? \nEarlier we mentioned paper laboratory notebooks, the most common recording device since scientific research began (although for sheer volume, it may have been eclipsed by computer hard drives). Have you ever wondered about the economics of laboratory notebooks? Cost is easy to understand, but the value of the data and information that is recorded there requires further explanation.\nThe value of the material recorded in a notebook depends on two key factors: the quality of the work and an inherent ability to put that documented work to use. The quality of the work is a function of those doing the work, how diligent they are, and the veracity of what has been written down. The inherent ability to use it depends upon the clarity of the writing, people\u2019s ability to understand it without recourse to the author, and access to the material. That last point is extremely important. Just by glancing at Figure 1, you can figure out where this is going.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Paper notebooks' cost vs. value, as a function of usage\n\n\n\nAs a scientist\u2019s notebook fills with entries, it gains value because of the content. Once filled, it reaches an upper limit and is placed in a library. There it takes a slight drop in value because its ease-of-access has changed; it isn\u2019t readily at hand. As library space fills, the notebooks are moved to secondary storage (in one company I worked at, secondary storage consisted of trailers in a parking lot). Costs go up due to the cost of owning or renting the secondary storage and the space they take. The object's value drops, not because of the content but due to the difficulty in retrieving that content (e.g., which trailer? which box?). Unless the project is still active, the normal turn-over of personnel (e.g., via promotions, movement around the company, leaving the company) mean that institutional memory diminishes and people begin to forget the work exists. If few researchers can remember it, find it, and access it, the value drops regardless of the resources that went into the work. That is compounded by the potential for physical deterioration of the object (e.g., water damage, mice, etc.).\nPreventing the loss of access to the results of your investment in R&D projects will rely on information technology. That reliance will be built upon planning an effective informatics environment, which is precisely where this discussion is going. How is putting you lab results into a computer system any different than a paper-based laboratory notebook? There are obvious things like faster searching and so on, but from our previous discussion on them, not much is different; you still have essentially a single point of failure, unless you plan for that eventuality. That is the fundamental difference and what will drive the rest of this writing: \n\nPlanning builds in reliability, security, and protection against loss. (Oh, and it allows us to work better, too!)\nYou could plan for failure in a paper-based system by making copies, but those copies still represent paper that has to be physically managed. With electronic systems, we can plan for failure by using automated backup procedures that make faithful copies, as many as we\u2019d like, at low cost. This issue isn\u2019t unique to laboratory notebooks, but it is a problem for organizations that depends on paper records.\nThe difference between writing on paper and using electronic systems isn\u2019t limited to how the document is realized. If you were to use a typewriter, the characters would show up on the paper and you'd be able to read them; all you needed was the ability to read (which could include braille formats) and understand what was written. However, if you were using a word processor, the keystrokes would be captured by software, displayed on the screen, placed in the computer\u2019s memory, and then written to storage. If you want to read the file, you need something\u2014software\u2014to retrieve it from storage, interpret the file contents, determine how to display it, and then display it. Without that software the file is useless. A complete backup process has to include the software needed to read the file, plus all the underlying components that it depends upon. You could correctly argue that the hardware is required as well, but there are economic tradeoffs as well as practical ones; you could transfer the file to other hardware and read it there for example. \nThat point brings us to the second subject of this writing: technology management. What do I have to do to make sure that I have the right tools to enable me to work? The problem is simple enough when all you're concerned with is writing and preparing accompanying graphics. Upon shifting the conversation to laboratory computing, it gets more complicated. Rather than being concerned with one computer and a few software packages, you have computers that acquire and process data in real-time[e], transmit it to other computers for storage in databases, and systems that control sample processing and administrative work. Not only do the individual computer systems and the equipment and people they support have to work well, but also they have to work cooperatively, and that is why we have to address planning and technology management in laboratory work.\nThat brings us to a consideration of what lab work is all about.\n\nDifferent ways of looking at laboratories \nWhen you think about a \u201claboratory,\u201d a lot depends on your perspective: are you on the outside looking in, do you work in a lab, or are you taking that high school chemistry class? When someone walks into a science laboratory, the initial impression is that of confusing collection of stuff, unless they're familiar with the setting. \u201cStuff\u201d can consist of instruments, glassware, tubing, robots, incubators, refrigerators and freezers, and even petri dishes, cages, fish tanks, and more depending on the kind of work that is being pursued.\nFrom a corporate point of view, a \"laboratory\" can appear differently and have different functions. Possible corporate views of the laboratory include:\n\n A laboratory is where questions are studied, which may support other projects or provide a source of new products, acting as basic and applied R&D. What is expected out of these labs is the development of new knowledge, usually in the form of reports or other documentation that can move a project forward.\n A laboratory acts as a research testing facility (e.g., analytical, physical properties, mechanical, electronics, etc.) that supports research and manufacturing through the development of new test methods, special analysis projects, troubleshooting techniques, and both routine and non-routine testing. The laboratory's results come in the form of reports, test procedures, and other types of documented information.\n A laboratory acts as a quality assurance\/quality control (QA\/QC) facility that provides routine testing, producing information in support of production facilities. This can include incoming materials testing, product testing, and product certification.\nTypically, stakeholders outside the lab are looking for some form of result that can be used to move projects and other work forward. They want it done quickly and at low cost, but also want the work to be of high quality and reliability. Those considerations help set the goals for lab operations.\nWithin the laboratory there are two basic operating modes or workflows: project-driven or task-driven work. With project-driven workflows, a project goal is set, experiments are planned and carried out, the results are evaluated, and a follow-up course of action is determined. This all requires careful documentation for the planning and execution of lab work. This can also include developing and revising standard operating procedures (SOPs). Task-driven workflows, on the other hand, essentially depend on the specific steps of a process. A collection of samples needs to be processed according to an SOP, and the results recorded. Depending upon the nature of the SOP and the number of samples that have to be processed, the work can be done manually, using instruments, or with partial or full automation, including robotics. With the exception of QA\/QC labs, a given laboratory can use a combination of these modes or workflows over time as work progresses and the internal\/external resources become available. QA\/QC labs are almost exclusively task-driven; contract testing labs are as well, although they may take on project-driven work.\nWithin the realm of laboratory informatics, project-focused work centers on the electronic laboratory notebook (ELN), which can be described as a lab-wide diary of work and results. Task-driven work is organized around the laboratory information management system (LIMS)\u2014or laboratory information system (LIS) in clinical lab settings\u2014which can be viewed as a workflow manager of tests to be done, results to be recorded, and analyses to be finalized. Both of these technologies replaced the paper-based laboratory notebook discussed earlier, coming with considerable improvements in productivity. And although ELNs are considerably more expensive than paper systems, the short- and long-term benefits of an ELN overshadow that cost issue.\n\nLabs in transition, from manual operation to modern facilities \n Figure 2. Chemical laboratory at Oswego SUNY 1893Laboratories didn\u2019t start with lots of electronic components; they began with people, lab benches, glassware, Bunsen burners, and other equipment. Lab operations were primarily concerned with peoples' ability to work. The technology was fairly simple by today\u2019s standards (Figure 2), and an individual\u2019s skills were the driving factor in producing quality results.\nFor the most part, the skills you learned in school were the skills you needed to be successful here as far as technical matters went; management education was another issue. That changed when electronic instrumentation became available. Analog instruments such as scanning spectrophotometers, chromatographs, mass spectrometer, differential scanning calorimeters, tensile testers, and so on introduced a new career path to laboratory work: the instrument specialist, who combined an understanding of the basic science with the an understanding of the instrument\u2019s design, as well as how to use it (and modify it where needed), maintain it, troubleshoot issues, and analyze the results. Specialization created a problem for schools: they couldn\u2019t afford all the equipment, find knowledgeable instructors, and encourage room in the curriculum for the expanding subject matter. Schools were no longer able to educate people to meet the requirements of industry and graduate-level academia. And then digital electronics happened. Computers first became attached to instruments, and then incorporated into the instrumentation.[f]\nThe addition of computer hardware and software to an instrument increased the depth of specialization in those techniques. Not only did you have to understand the science noted above, but also the use of computer programs used to work with the instrument, how to collect the data, and how to perform the analysis. An entire new layer of skills was added to an already complex subject.\nThe latest level of complexity added to laboratory operations has been the incorporation of LIMS, ELNs, scientific data management systems (SDMS), and laboratory execution systems (LES) either as stand-alone modules or combined into more integrated packages or \"platforms.\"\n\nThere's a plan for that? \nIt is rare to find a lab that has an informatics plan or strategy in place before the first computer comes through the door; those machines enter as part of an instrument-computer control system. Several computers may use that route to become part of the lab's technology base before people realize that they need to start taking lab computing seriously, including how to handle backups, maintenance, support, etc.\nFirst computers come into the lab, and then the planning begins, often months later, as an incremental planning effort, which is the complete reverse of how things need to be developed. Planning is essential as soon as you decide that a lab space will be created. That almost never happens, in part because no one has told you that is required, let alone why or how to go about it.\n\nThinking about a model for lab operations \nThe basic purpose of laboratory work is to answer questions. \u201cHow do we make this work?\u201d \u201cWhat is it?\u201d \u201cWhat\u2019s the purity of this material?\u201d These questions and others like them occur in chemistry, physics, and the biological sciences. Answering those questions is a matter of gathering data and information through observation and experimental work, organizing it, analyzing it, and determining the next steps needed as the work moves forward (Figure 3). Effective organization is essential, as lab personnel will need to search data and information, extract it, move it from one data system to another for analysis, make decisions, update planning, and produce interim and ultimately final reports.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Simplified flow of data\/information from sources to collection system\n\n\n\nOnce the planning is done, scientific work generally begins with collecting observations and measurements (Data\/Information Sources 1\u20134, Figure 3) from a variety of sources. Lab bench work usually involves instrumentation, and many instruments have computer controls and data systems as part of them. This is the more visible part of lab work and the one that matches people\u2019s expectations for a \u201cscientific lab.\u201d This is where most of the money is spent on equipment, materials, and people\u2019s expertise and time. All that expenditure of resources results in \u201cthe pH of the glowing liquid is 6.5,\u201d \u201cthe concentration of iron in the icky stuff is 1500 ppm,\u201d and so on. That\u2019s the end result of all those resources, time, and effort put into the scientific workflow. That\u2019s why you built a million-dollar facility (in some spheres of science such as astronomy, high energy physics, and the space sciences, the cost of collection is significantly higher). So what do you do with those results? Prior to the 1970s, the collection points were paper: forms, notebooks, and other document, all with their earlier discussed issues.\nThe material on those instrument data systems needs to be moved to an intermediate system for long-term storage and reference (the second step of Figure 3). This is needed because those initial data systems may fail, be replaced, or added to as the work continues. After all, the data and information they\u2019ve collected needs to be preserved, organized, and managed to support continued lab work.\nThe analyzed results need to be collected into a reference system that is the basis of long-term analysis, management\/administration work, and reporting. This last system in the flow is the central hub of lab activities; it is also the distribution point for material sent to other parts of the organization (the third and fourth stages of Figure 3). While it is natural for scientists to focus on the production of data and information, the organization and centralized management of the results of laboratory work needs to be a primary consideration. That organization will be focused of short- and long-term data analysis and evaluation. The results of this get used to demonstrate the lab's performance towards meeting its goals, and it will show those investing in your work that you\u2019ve got your management act together, which is useful when looking for continued support.\nToday, those systems come in two basic forms: LIMS and ELN. The details of those systems are the subject of a number of articles and books.[7] Without getting into too much detail:\n\n LIMS are used to support testing labs managing sample workflows and planning, as well as cataloging results (e.g., short text and numerical information).\n ELNs are usually found in research functioning as an electronic diary of lab work for one or more scientists and technicians. The entries may contain extensive textural material, numerical entries, charts, graphics, etc. The ELN is generally more flexible than a LIMS.\nThat distinction is simplistic; some labs support both activities and need both types of systems, or even a hybrid package. However, the description is sufficient to get us to the next point: the lifespan of systems varies, depending on where you are looking in Figure 3's model. Figure 4 gives a comparison.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. The relative lifespan of laboratory systems\n\n\n\nThe experimental methods\/procedures used in lab work will change over time as the needs of the lab change. Older instruments may be updated and new ones introduced. Retirement is a problem, particularly if data systems are part of the equipment. You have to have access to the data. That need will live on long past the equipment's life. That is one reason that moving data and information to an intermediate system like an SDMS is important. However, in some circumstances, even that isn\u2019t going to be sufficient (regulated industries where the original data structures and software that generated them need to be preserved as an operating entity). In those cases, you may have old computers stacked up just in case you need access to their contents. A better way is to virtualize the systems as containers on servers that support a virtualized environment.\nVirtualization\u2014making an electronic copy of computer system and running on a server\u2014is potentially a useful technology in lab work; while it won\u2019t participate in day-to-day activities it does have a role. Suppose you have an instrument-data system that is being replaced or retired. Maybe the computer is showing signs of aging or failing. What do you do with the files and software that are on the computer portion of the combination? You can\u2019t dispose of them because you may need access to those data files and software later. On the other hand, do you really want to collect computer systems that have to be maintained just to have access to the data if and when you need it? Instead, virtualization is a software\/hardware technology that allows you to make a complete copy of everything that is on that computer\u2014including operating system files, applications, and data files\u2014and stores it in one big file referred to as a \u201ccontainer.\u201d That container can be moved to a computer that is a virtual server and has software that emulates various operating environment, allowing the software in the container to run as if it were on its own computer hardware. A virtual server can support a lot of containers, and the operating systems in those containers can be updated as needed. The basic idea is that you don\u2019t need access to a separate physical computer; you just need the ability to run the software that was on it. If your reaction to that is one of dismay and confusion, it\u2019s time to buy your favorite IT person a cup of coffee and have a long talk. We\u2019ll get into more details when we cover data backup issues.\nWhy is this important to you?\nWhile the science behind producing results is the primary reason your lab exists, gaining the most value from the results is essential to the organization overall. That value is going to be governed by the quality of the results, ease of access, the ability to find and extract needed information easily, and a well-managed K\/I\/D architecture. All of that addresses a key point from management\u2019s perspective: return on investment or ROI. If you can demonstrate that your data systems are well organized and maintained, and that you can easily find and use the results from experimental work and contribute to advancing the organization\u2019s goals, you\u2019ll make it easier to demonstrate solid ROI and gain funding for projects, equipment, and people needed to meet your lab's goals.\n\r\n\n\nThe seven goals of planning and managing lab technologies \nThe preceding material described the need for planning and managing lab technologies, and making sure lab personnel are qualified and educated to participate in that work. The next step is the actual planning. There are at least two key aspects to that work: planning activities that are specific and unique to your lab(s) and addressing broader scope issues that are common to all labs. The discussion found in the rest of this guide is going to focus on the latter points.\nEffective planning is accomplished by setting goals and determining how you are going to achieve them. The following sections of this guide look at those goals, specifically:\n\n Supporting an environment that fosters productivity and innovation\n Developing high-quality data and information\n Managing knowledge, information, and data effectively, putting them in a structure that encourages use and protects value\n Ensuring a high level of data integrity at every step\n Addressing security throughout the lab \n Acquiring and developing \"products\" that support regulatory requirements\n Addressing systems integration and harmonization\nThe material below begins the sections on goal setting. Some of these goals are obvious and understandable, others like \u201charmonization\u201d are less so. The goals are provided as an introduction rather than an in-depth discussion. The intent is to offer something suitable for the purpose of this material and a basis for a more detailed exploration at a later point. The intent of these goals is not to tell you how to do things, but rather what things need to be addressed. The content is provided as a set of questions that you need to think about. The answers aren't mine to give, but rather yours to develop and implement; it's your lab. In many cases, developing and implementing those answers will be a joint effort by all stakeholders.\n\nFirst goal: Support an environment that fosters productivity and innovation \nIn order to successfully plan for and manage lab technologies, the business environment should ideally be committed to fostering a work environment that encourages productivity and innovation. This requires:\n\n proven, supportable workflow methodologies;\n educated personnel;\n fully functional, inter-departmental cooperation;\n management buy-in; and\n systems that meet users' needs.\nThis is one of those statements that people tend to read, say \u201csure,\u201d and move on. But before you do that, let\u2019s take a look at a few points. Innovation may be uniquely human (not even going to consider AI), and the ability to be \u201cinnovative\u201d may not be universal.\nPeople need to be educated, be able to separate true facts from \u201cbeliefs,\u201d and question everything (which may require management support). Innovation doesn\u2019t happen in a highly structured environment, you need the freedom to question, challenge, etc. You also need the tools to work with. The inspiration that leads to innovation can happen anywhere, anytime. All of a sudden all the pieces fit. And then what? That is where a discussion of tools and this work come together.\nIf a sudden burst of inspiration hits, you want to do it now and not after traveling to an office, particularly if it is weekend or vacation. You need access to knowledge (e.g., documents, reports), information, and data (K\/I\/D). In order to do that, a few things have to be in place:\n\n Business and operations K\/I\/D must be accessible.\n Systems security has to be such that a qualified user can gain access to K\/I\/D remotely, while preventing its unauthorized use.\n Qualified users must have the hardware and software tools required to access the K\/I\/D, work with it, and transmit the results of that work to whoever needs to see it.\n Qualified users must also be able to remotely initiate actions such as testing.\nThose elements depend on a well-designed laboratory and corporate informatics infrastructure. Laboratory infrastructure is important because that is where the systems are that people need access to, and corporate infrastructure is important since corporate facilities have to provide access, controls, and security. Implementation of those corporate components has to be carefully thought through; they must be strong enough to frustrate unwarranted access (e.g., multi-factor logins) while allowing people to get real work done.\nAll of this requires flexibility and trust in people, an important part of corporate culture. This will become more important as society adjusts to new modes of working (e.g., working online due to a pandemic) and the realization that the fixed format work week isn\u2019t the only way people can be productive. For example, working from home or off-site is increasingly commonplace. Laboratory professionals work in two modes: intellectual, which can be done anywhere, and the lab bench, where physical research tasks are performed. We need to strike a balance between those modes and the need for in-person vs virtual contact.\nLet's take another look at the previous Figure 3, which offered one possible structure for organizing lab systems:\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Simplified flow of data\/information from sources to collection system\n\n\n\nThis use of an intermediate file storage system like an SDMS and the aggregation of some instruments to a common computer (e.g., one chromatographic data system for all chromatographs vs. one per instrument) becomes more important for two reasons: 1. it limits the number of systems that have to be accessed to search, organize, extract, and work with K\/D\/I, and 2. it makes it easier to address security concerns. There are additional reasons why this organization of lab systems is advantageous, but we\u2019ll cover those in later installments. The critical point here is a sound informatics architecture is key to supporting innovation. People need access to tools and K\/D\/I when they are working, regardless of where they are working from. As such, those same people need to be well-versed in the capabilities of the systems available to them, how to access them, use them, and how to recognize \u201cmissing technologies,\u201d capabilities they need but don\u2019t have access to or simply don't exist.\nImagine this. A technology expert consults for two large organizations, one tightly controlled (Company A), the other with a liberal view of trusting people to do good work (Company B). In the first case, getting work done can be difficult, with the expert fighting through numerous reviews, sign-offs, and politics. Company A has a stated philosophy that they don\u2019t want to be the first in the market with a new product, but would rather be a strong number two. They justify their position through the cost of developing markets for new products: let someone else do the heavy lifting and follow behind them. This is not a culture that spawns innovation. Company B, however, thrives on innovation. While processes and procedures are certainly in place, the company has a more relaxed philosophy about work assignments. If the expert has a realizable idea, Company B lets them run with it, as long as they complete their assigned workload in a timely fashion. This is what spurs the human side of innovation.\n\nSecond goal: Develop high-quality data and information \nAsking staff to \"develop high-quality data and information\" seems like a pretty obvious point, but this is where professional experience and the rest of the world part company. Most of the world treats \u201cdata\u201d and \u201cinformation\u201d as interchangeable words. Not here.\nThere are three key words that are going to be important in this discussion of goals: knowledge, information, and data (K\/I\/D). We\u2019ll start with \u201cknowledge\u201d. The type of knowledge we will be looking at is at the laboratory\/corporate level, the stuff that governs how a laboratory operates, including reports, administrative material, and most importantly standard operating procedures (SOPs). SOPs tell us how lab work is carried out via its methods, procedures, etc. (This subject parallels the topic of \u201cdata integrity,\u201d which will be covered later.) Figure 5 positions K\/I\/D with respect to each other within laboratory processes.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Simplified flow of data\/information from sources to collection system\n\n\n\nThe diagram in Figure 5 is a little complicated, and we\u2019ll get into the details as the material develops. For the moment, we\u2019ll concentrate on the elements in black.\nAs noted above, SOPs guide activities within the lab. As work is defined\u2014both research and testing\u2014SOPs have to be developed so that people know how to carry out their tasks consistently. Our first concern then is proper management of SOPs. Sounds simple, but in practice it isn\u2019t. It\u2019s a matter of first developing and updating the procedures, documenting them, and then managing both the documents and the lab personnel using them.\nWhen developing, updating, and documenting procedures, a lab will primarily be looking at the science its working with and how regulatory requirements affect it, particularly in research environments. Once developed, those procedures will eventually need to be updated. But why is an update to a procedure needed? What will the effects of the update be based on the changes that were made, and how do the results of the new version compare to the previous version? That last point is important, and to answer it you need a reference sample that has been run repeatedly under the older version so that you have a solid history of the results (i.e., control chart) over time. You also need the ability to run that same reference sample under the new procedure to show that there are no differences, or that differences can be accounted for. If differences persist, what do you do about the previous test results under the old procedure?\nThe idea of running one or more stable reference samples periodically is a matter of instituting statistical process control over the analysis process. It can show that a process is under control, detect drift in results, and demonstrate that the lab is doing its job properly. If multiple analysts are doing the same work, it can also reveal how their work compares and if there are any problems. It is in effect looking over their shoulders, but that just comes with the job. If you find that the amount of reference material is running low, then phase in a replacement, running both samples in parallel to get a documented comparison with a clean transition from one reference sample to another. It\u2019s a lot of work and it\u2019s annoying, but you\u2019ll have a solid response when asked \u201care you confident in these results?\u201d You can then say, \u201cYes, and here is the evidence to back it up.\u201d\nAfter the SOPs have been documented, they must then be effectively managed and implemented. First, take note of the education and experience required for lab personnel to properly implement any SOP. Periodic evaluation (or even certification) would be useful to ensure things are working as they should. This is particularly true of procedures that aren\u2019t run often, as people may forget things. \nAnother issue of concern with managing SOPs is how to manage versioning. Consider two labs. Lab 1 is a well-run lab. When a new procedure is issued, the lab secretary visits each analyst, takes their copy of\nthe old method, destroys it, provides a copy of the new one, requires the analyst sign for receipt, and later requires a second signature after the method has been reviewed and understood. Additional education is also provided on an as-needed basis. Lab 2 has good intentions, but it's not as proactive as Lab 1. Lab 2 retains all documents on a central server. Analysts are able to copy a method to their machines and use it. However, there is no formalized method of letting people know when a new method is released. At any given time there may be several analysts running the same method using different versions of the related SOP. The end result is having a mix of samples run by different people according to different SOPs. \nThis comparison of two labs isn\u2019t electronic versions vs. paper, but rather a formal management structure vs. a loose one. There\u2019s no problem maintaining SOPs in an electronic format, as there are many benefits, but there shouldn\u2019t be any question about the current version, and there should be a clear process for notifying people about updates while also ensuring that analysts are currently educated in the new method's use.\nManaging this set of problems\u2014analyst education, versions of SOPs, qualification of equipment, current reagents, etc.\u2014 was the foundation for one of the early original ELNs, SmartLab by Velquest, now developed as a LES by Dassault Syst\u00e8mes as part of the BIOVIA product line. And while Dassault's LES, and much of the Biovia product line, narrowly focuses on their intended market, the product remains suitable for any lab where careful control over procedure execution is warranted. This is important to note, as a LES is designed to guide a person through a procedure from start to finish, making it one step away from engaging in a full robotics system (robotics may play a role in stages of the process). The use of an LES doesn\u2019t mean that personnel aren\u2019t trusted or deemed incompetent; rather, it is a mechanism for developing documented evidence that methods have been executed correctly. That evidence builds confidence in results.\nLESs are available from several vendors, often as part of their LIMS or ELN offerings. Using any of these systems requires planning and scripting (a gentler way of saying \u201cprogramming\u201d), and the cost of implementation has to be balanced against the need (does the execution of a method require that level of sophistication) and ROI.\nUp to this point, we\u2019ve looked at developing and managing SOPs, as well as at least one means of controlling experiment\/procedure execution. However, there are other ways of going about this, including manual and full robotics systems. Figure 6 takes us farther down the K\/I\/D model to elaborate further on experiment\/procedure execution.[g]\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. K\/I\/D flow diagram focusing on experiment\/procedure execution\n\n\n\nAs we move from knowledge development and management (i.e., SOPs), and then on to sample preparation (i.e., pre-experiment), the next step is usually some sort of measurement by an instrument, whether it is pH meter or spectrometer, yielding your result. That brings us to two words we noted earlier: \"data\" and \"information.\" We'll note the differences between the two using a gas chromatography system as an example (Figure 7), as it and other base chromatography systems are among the most widely used of upper-tier instrumentation and widely found in labs where chemical analysis is performed.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 7. Gas chromatography \"data\" and \"information\"\n\n\n\nAs we look at Figure 7, we notice to the right of the vertical blue line is an output signal from a gas chromatograph. This is what chromatographers analyzed and measured when they carried out their work. The addition of a computer made life easier by removing the burden of calculations, but it also added complexity to the work in the form of having to manage the captured electronic data and information. An analog-to-digital (A\/D) converter transformed those smooth curves to a sequence of numbers that are processed to yield parameters that described the peaks, which in turn were used to calculate the amount of substance in the sample. Everything up to that last calculation\u2014left of the vertical blue line\u2014is \u201cdata,\u201d a set of numerical values that, taken individually, have no meaning by themself. It is only when we combine it with other data sets that we can calculate a meaningful result, which gives us \u201cinformation.\u201d\nThe paragraph above describes two different types of data:\n1. the digitized detector output or \"raw data,\" constituting a series of readings that could be plotted to show the instrument output; and\n2. the processed digitized data that provides descriptors about the output, with those descriptors depending largely upon the nature of the instrument (in the case of chromatography, the descriptors would be peak height, retention time, uncorrected peak area, peak widths, etc.).\nBoth are useful and neither of them should be discarded; the fact that you have the descriptors doesn\u2019t mean you don\u2019t need the raw data. The descriptors are processed data that depends on user-provided parameters. Changing the parameter can change the processing and the values assigned to those descriptors. If there are accuracy concerns, you need the raw data as a backup. Since storage is cheap, there really isn\u2019t any reason to discard anything, ever. (And in some regulatory environments, keeping raw data is mandated for a period of time.)\nIf you want to study the data and how it was processed to yield a result, you need more data, specifically the reference samples (standards) used to evaluate each sample. An instrument file by itself is almost useless without the reference material run with that sample. Ideally, you\u2019d want a file that contains all the sample and reference data that was analyzed in one session. That might be a series of manual samples analyzed or an entire auto-sampler tray.\nEverything we've discussed here positively contributes to developing high-quality data and information. When methods are proven and you have documented evidence that they were executed by properly educated personnel using qualified reagents and instruments, you then have the instrument data to support each sample result and any other information gleaned from that data.\nYou might wonder what laboratorians did before computers. They dealt with stacks of spectra, rolls of chromatograms, and laboratory notebooks, all on paper. If they wanted to find the data (e.g., a pen trace on paper) for a sample, they turned to the lab's physical filing system to locate it.[h] Why does this matter? That has to do with our third goal.\n\nThird goal: Manage K\/I\/D effectively, putting them in a structure that encourages use and protects value \nIn the previous section we introduced three key elements of laboratory work: knowledge, information, and data (K\/I\/D). Each of these is \u201cdatabase\u201d structures (\u201cdata\u201d in the general sense). We also looked at SOP management as an example of knowledge management, and distinguished \u201cdata\u201d and \u201cinformation\u201d management as separate but related concerns. We also introduced flow diagrams (Figures 5 and 6) that show the relationship and development of each of those elements.\nIn order for those elements to justify the cost of their development, they have to be placed in systems that encourage utilization and thus retain their value. Modern informatics tools assist in many ways:\n\n Document management systems support knowledge databases (and some LIMS and ELNs inherently support document management).\n LIMS and ELNs provide a solid base for laboratory information, and they may also support other administrative and operational functions.\n Instrument data systems and SDMS collect instrument output in the form of reports, data, and information.\nYou may notice there is significant functional redundancy as vendors try to create the \u201cultimate laboratory system.\u201d Part of lab management\u2019s responsibility is to define what the functional architecture should look like based on their current and perceived needs, rather than having it defined for them. It\u2019s a matter of knowing what is required and seeing what fits rather than fitting requirements into someone else\u2019s idea of what's needed.\nManaging large database systems is only one aspect of handling K\/I\/D. Another aspect involves the consideration of cloud vs. local storage systems. What option works best for your situation, is the easiest to manage, and is supported by IT? We also have to address the data held in various desktop and mobile computing devices, as well as bench top systems like instrument data systems. There are a number of considerations here, not the least of which is product turnover (e.g., new systems, retired systems, upgrades\/updates, etc.). (Some of these points will be covered latter on in other sections.)\nWhat you should think about now is the number of computer systems and software packages that you use on a daily basis, some of which are connected to instruments. How many different vendors are involved? How big are vendors (e.g., small companies\/limit staff, large organizations)? How often do they upgrade their systems? What\u2019s the likelihood they\u2019ll be around in two or five years?\nAlso ask what data file formats the vendor uses; these formats vary widely among vendors. Some put everything in CSV files, others in proprietary formats. In the latter case, you may not be able to use the data files without the vendor's software. In order to maintain the ability to work with instrument data, you will have to manage the software needed to open files and work with it, in addition to just making sure you have copies of the data files. In short, if you have an instrument-computer combination that does some really nice stuff and you want to preserve the ability to gain value from that instrument's data files, you have to make a backup copy of the software environment and the data files. This is particularly important if you're considering retiring a system that you'll still want to access data from, plus you may have to maintain any underlying software license. This is where the previous conversation about virtualization and containers comes in.\nIf you think about a computer system it has two parts: hardware (e.g., circuit boards, hard drive, memory, etc.) and software (e.g., the OS, applications, data files, etc.). From the standpoint of the computer\u2019s processor, everything is either data or instructions read from one big file on the hard drive, which the operating system has segmented for housing different types of files (that segmentation is done for your convenience; the processor just sees it all as a source of instructions and data). Virtualization takes everything on the hard drive, turns it into a complete file, and places that file onto a virtualization server where it is stored as a file called a \u201ccontainer.\u201d That server allows you to log in, open a container, and run it as though it were still on the original computer. You may not be able to connect it the original instruments to the containerized environment, but all the data processing functions will still be there. As such, a collection of physical computers can become a collection of containers. An added benefit of virtualizations applies when you're worried about an upgrade creating havoc with your application; instead, make a container as a backup.[i]\nThe advantage of all this is that you continue to have the ability to gain value and access to all of your data and information even if the original computer has gone to the recycle bin. This of course assumes your IT group supports virtualization servers, which provide an advantage in that they are easier to maintain and don\u2019t take up much space. In larger organization this may already be happening, and in smaller organizations a conversation may be had to determine IT's stance. The potential snag in all this is whether or not the software application's vendor license will cover the operation of their software on a virtual server. That is something you may want to negotiate as part of the purchase agreement when you buy the system.\nThis section has shown that effective management of K\/I\/D is more than just the typical consideration of database issues, system upgrades, and backups. You also have to maintain and support the entire operating system, the application, and the data file ecosystem so that you have both the files needed and the ability to work with them.\n\nFourth goal: Ensure a high level of data integrity at every step \n\u201cData integrity\u201d is an interesting couple of words. It shows up in marketing literature to get your attention, often because it's a significant regulatory concern. There are different aspects to the topic, and the attention given often depends on a vendor's product or the perspective of a particular author. In reality, it touches on all areas of laboratory work. The following is an introduction to the goal, with more detail given in later sections.\n\nDefinitions of data integrity \nThere are multiple definitions of \"data integrity.\" A broad encyclopedic definition can be found at Wikipedia, described as \"the maintenance of, and the assurance of, data accuracy and consistency over its entire life-cycle\" and \"a critical aspect to the design, implementation, and usage of any system that stores, processes, or retrieves data.\"[8]\nAnother definition to consider is from a more regulatory perspective, that of the FDA. In their view, data integrity focuses on the completeness, consistency, accuracy, and validity of data, particularly through a mechanism called the ALCOA+ principles. This means the data should be[9]:\n\n Attributable: You can link the creation or alteration of data to the person responsible.\n Legible: The data can be read both visually and electronically.\n Contemporaneous: The data was created at the same time that the activity it relates to was conducted.\n Original: The source or primary documents relating to the activity the data records are available, or certified versions of those documents are available, e.g., a notebook or raw database. (This is one reason why you should collect and maintain as much data and information from an instrument as possible for each sample.)\n Accurate: The data is free of errors, and any amendments or edits are documented.\nPlus, the data should be:\n\n Complete: The data must include all related analyses, repeated results, and associated metadata.\n Consistent: The complete data record should maintain the full sequence of events, with date and time stamps, such that the steps can be repeated.\n Enduring: The data should be able to be retrieved throughout its intended or mandated lifetime.\n Available: The data is able to be accessed readily by authorized individuals when and where they need it.\nBoth definitions revolve around the same point: the data a lab produces has to be reliable. The term \"data integrity\" and its associated definitions are a bit misleading. If you read the paragraphs above you get the impression that the focus in on the results of laboratory work, when in fact it is about every aspect of laboratory work, including the methods used and those who conduct those methods.\nIn order to gain meaningful value from laboratory K\/I\/D, you have to be assured of its integrity; \u201cthe only thing worse than no data, is data you can\u2019t trust.\u201d That is the crux of the matter. How do you build that trust? Building a sense of confidence in a lab's data integrity efforts requires addressing three areas of concern and their paired intersections: science, people, and informatics technology. Once we have successfully managed those areas and intersection points, we are left with the intersection common to all of them: constructed confidence in a laboratory's data integrity efforts (Figure 8). \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 8. The three areas contributing to data integrity and how they intersect\n\n\n\n Figure 9. The \"Science\" component\nThe science\nWe\u2019ll begin with a look at the scientific component of the conversation (Figure 9). Regardless of the kinds of questions being addressed, the process of answering them is rooted in methods and procedures. Within the context of this guide, those methods have to be validated or else your first step in building confidence has failed. If those methods end with electronic measurements, then that equipment (including settings, algorithms, analysis, and reporting) have to be fully understood and qualified for use in the validated process. The manufacturer's default settings should either be demonstrated as suitable or avoided.\n\r\n\nThe people\n\n Figure 10. The \"People\" componentPeople (Figure 10) need to be thoroughly educated and competent to meet the needs of the laboratory's operational procedures and scientific work. That education needs to extend beyond the traditional undergraduate program and include the specifics of instrumental techniques used. A typical four-year program doesn\u2019t have the time to cover the basic science and the practical aspects of how science is conducted in modern labs, and few schools can afford the equipment needed to meet that challenge. This broader educational emphasis is part of the intersection of people and science.\nAnother aspect of \u201cpeople\u201d is the development of a culture that contributes to data integrity. Lab personnel need to be educated on the organization\u2019s expectations of how lab work needs to be managed and maintained. This includes items such as records retention, dealing with erroneous results, and what constitutes original data. They should also be fully aware of corporate and regulatory guidelines and the effort needed to enforce them.[j] This is another instance where education beyond that provided in the undergraduate curriculum is needed.\n\r\n\nInformatics technology\n\n Figure 11. The \"Informatics Technology\" componentLaboratory informatics technology (Figure 11) is another area where data integrity can either be enhanced or lost. The lab's digital architecture needs to be designed to support relatively easy access (within the scope of necessary security considerations) to the lab's data from the raw digitized detector output, through intermediate processed stages and to the final processed information. Unnecessary duplication of K\/D\/I must be avoided. You also need to ensure that the products chosen for lab work are suitable for the work and have the ability to be integrated electronically. After all, the goal is to avoid situations where the output of one system is printed and then manually entered into another.\nThe implementation and use of informatics technology should be the result of careful product selection and their intentional design\u2014from the lab bench to central database systems such as LIMS, ELN,\nSDMS, etc.\u2014rather than haphazard approach of an aggregate of lab computers.\nOther areas of concern with informatics technology include backups, security, and product life cycles, which will be addressed in later sections. If as we continue onward through these goals it appears like everything touches on data integrity, it's because it does. Data integrity can be considered an optimal result of the sum of well-executed laboratory operations.\n\r\n\nThe intersection points\nTwo of the three intersection points deserve minor elaboration (Figure 12). First, the intersection of people and informatics technologies has several aspects the address. The first is laboratory personnel\u2019s responsibility\u2014which may be shared with corporate or LAB-IT\u2014for the selection and management of informatics products. The second is the fact that this requires those personnel to be knowledgeable concerning the application of informatics technologies in laboratory environments. Ensure the selected personnel have the appropriate backgrounds and knowledge to consider, select, and effectively use those products and technologies.\nThe other intersection point to be addressed is that of science with informatics technology. Here, stakeholders are concerned with product selection, system design (for automated processes), and system integration and communication with other systems and instruments. Again, as noted above, we go into more detail in later sections. The primary point here, however, can be summed up as determining whether or not the products selected for your scientific endeavors are compatible with your data integrity goals.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 12. The intersection of People and Informatics Technology (left) and Science and Informatics Technology (right)\n\n\n\nAddressing the needs of these two intersection points requires deliberate effort and many planning questions regarding vendor support, quality of design, system interoperability, result output, and scientific support mechanisms. Questions to ask include:\n\n Vendor support: How responsive are vendors to product issues? Do you get a fast, usable response or are you left hanging? A product that is having problems can affect data quality and reliability.\n Quality of design: How easy is the system to use? Are controls, settings, and working parameters clearly defined and easily understood? Do you know what effect changes in those points will have on your results? Has the system been tuned to your needs (not adjusted to give you the answers you want, but set to give results that truly represent the analysis)? Problems with adjusting settings properly can distort results. (This is one area where data integrity may be maintained throughout a process, and then lost because of improper or untested controls on an instrument's operation.)\n System interoperability: Will there be any difficulty in integrating a software product or instrument into a workflow? Problems with sample container compatibility, operation, control software, etc. can cause errors to develop in the execution of a process flow. For example, problems with pipette tips can cause errors in fluid delivery.\n Result output: Is an electronic transfer of data possible, or does the system produce printed output (which means someone typing results into another system)? How effective is the communications protocol; is it based on a standard or does it require custom coding, which could be error prone or subject to interference? Is the format of the data file one that prevents changes to the original data? For example, CSV files allow easy editing and have the potential for corruption, nullifying data integrity efforts.\n Scientific support mechanisms: Does the product fully meet the intended need for functionality, reliability, and accuracy?\nThe underlying goal in this section goes well beyond the material that is covered in schools. Technology development in instrumentation and the application of computing and informatics is progressing rapidly, and you can\u2019t assume that everything is working as advertised, particularly for your application. Software has bugs and hardware has limitations. Applying healthy skepticism towards products and requiring proof that things work as needed protect the quality of your work. \nIf you\u2019re a scientist reading this material, you might wonder why you should care. The answer is simply this: it is the modern evolution of how laboratory work gets done and how results are put to use. If you don\u2019t pay attention to the points noted, data integrity may be compromised. You may also find yourself the unhappy recipient of a regulatory warning letter.\nWhile there are some outcomes that could occur that you prefer didn't, there are also positive outcomes to come from your data integrity efforts: your work will be easier and protected from loss, results will be easier to organize and analyze, and you\u2019ll have a better functioning lab. You\u2019ll also have fewer unpleasant surprises when technology changes occur and you need to transition from one way of doing things to another. Yet there's more to protecting the integrity of your K\/I\/D than addressing the science, people, and information technology of your lab. The security of your lab and its information systems must also be addressed.\n\nFifth goal: Addressing security throughout the lab \nSecurity is about protection, and there are two considerations in this matter: what are we protecting and how do we enact that protection? The first is easily answered by stating that we're protecting our ability to effectively work, as well as the results of that work. This is largely tied to the laboratory's data integrity efforts. The second consideration, however, requires a few more words.\nBroadly speaking, security is not a popular subject in science, as it is viewed as not advancing scientific work or the development of K\/I\/D. Security is often viewed as inhibiting work by imposing a behavioral structure on people's freedom to do their work how they wish. Given these perceptions, it should be a lab's goal to create a functional security system that provides the protection needed while at the same time minimizing the intrusion in people\u2019s ability to work.\nThis section will look at a series of topics that address the physical and electronic security of laboratory work. Those major topics are shown in Figure 13 below. The depth of the commentary will vary, with some topics getting discussed at length and others by brief reference to others' work.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 13. The key issues of laboratory security\n\n\n\nWhy must security be addressed in the laboratory? There are many reasons, which are best diagramed, as seen in Figure 14:\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 14. The primary reasons why security issues need to be addressed in the lab\n\n\n\nAll of these reasons have one thing in common: they affect our ability to work and access the results of that work. This requires a security plan. In the end, implemented security efforts either preserve those abilities, or they reduce the value and utility of the work and results, particularly if security isn't implemented well or adds a burden to personnel's ability to work. While addressing these reasons and their corresponding protections, we should keep in mind a number of issues when developing and implementing a security plan within the lab (Figure 15). Issues like remote access have taken on particular significance over the course of the COVID-19 pandemic.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 15. Issues to keep in mind when planning security programs\n\n\n\nWhen the subject of security comes up, people's minds usually go in one of two directions: physical security (i.e., controlled access) and electronic security (i.e., malware, viruses, ransomware, etc.). We\u2019re going to come at it from a different angle: how do the people in your lab want to work? Instead of looking at a collection of solutions to security issues, we\u2019re going to first consider how lab personnel want to be working and within what constraints, and then we'll see what tools can be used to make that possible. Coming at security from that perspective will impact the tools you use and their selection, including everything from instrument data systems to database products, analytical tools, and cloud computing. The lab bench is where work is executed, and the planning and thinking take place between our ears, something that can happen anywhere. How do we provide people with the freedom to be creative and work effectively (something that may be different for each of us) while maintaining a needed level of physical and intellectual property security? Too often security procedures seem to be designed to frustrate work, as noted in the previous Figure 15.\nThe purpose of security procedures are to protect intellectual property, data integrity, resources, our ability to work, and lab personnel, all of which can be impacted by the reasons given in the prior Figure 14. However, the planning for how to approach these security procedures requires the coordination with and cooperation of several stakeholders within and tangentially related to the laboratory. Ensure these and any other necessary stakeholders are involved with the security planning efforts of your laboratory:\n\n Facilities management: These stakeholders manage the physical infrastructure you are working in and have overall responsibility for access control and managing the human security assets in larger companies. In smaller companies and startups, the first line of security may be the receptionist; how well trained are they to deal with the subject?\n IT groups: These stakeholders will be responsible for designing and maintaining (along with facilities management) the electronic security systems, which range from passkeys to networks.\n Legal: These stakeholders may work with human resources to set personnel standards for security, reviewing licensing agreements and contracts\/leases for outside contractors and buildings (more later).\n Lab personnel: From the standpoint of this guide, this is all about the people doing the analytical and research work within the laboratory.\n Consultants: Security is a complex and rapidly developing subject, and you will likely need outside support to advise you on what is necessary and possible, as well as how to go about making that a reality.\nBut what else must be considered during you and your stakeholders' planning efforts? Before we can get into the specific technologies and practices that may be implemented within a facility, we need to look at the facility itself. \n\nExamine aspects of the facility itself \nDoes your company own the building you are working in? Is it leased? Is it shared with other companies in a single industrial complex? If you own the facility, life is simpler since you control everything. Working in a shared space that is leased or rented requires more planning and thought, preferably before you sign an agreement. You're likely to have additional aspects to seriously consider about your facility. Have the locks and door codes been changed since the last tenant left? Is power shared across other businesses in your building? Is the backup generator\u2014if there is one\u2014sufficient to run your systems? What fire protections are in place? How is networking managed in the facility? Are security personnel attuned to the needs of your company? Let's take a look at some of these and other questions that should be addressed.\nIs the physical space well-defined, and does building maintenance have open access to your various spaces? \nBuilding codes vary from place to place. Some are very specific and strict, while others are almost live-and-let-live. One thing you want to be able to do is to define and control your organization's physical space and set up any necessary and additional protective boundaries. Physical firewalls are one way of doing that. A firewall should be a solid structure that acts as a barrier to fire propagation between your space and neighboring spaces, extending from below-ground areas to the roof. If it is a multi-level structure, the levels should be isolated. This may seems obvious, but in some single-level shared buildings (e.g., strip malls) the walls may not go to the roof to make it easier to route utilities like HVAC, power, and fire suppression. This can acts as an access point to your space.\nBuilding maintenance is another issue. Do they have access to your space? Does that access come with formal consent or is that consent assumed as part of the lease or rental agreement? Several problems must be considered. First, know that anyone who has access to your physical space should be considered a weak point in your security. Employees should inherently have a vested interest in protecting your assets, but building maintenance is a different matter. Who vets them? Since these notes are focused on laboratory systems, who trains them about what to touch and what not to? (For example, an experiment could be ruined because maintenance personnel opened a fume hood, disturbing the airflow, despite any signage placed on the hood glass.) Consider more than just office systems in your space analysis, including other equipment that may be running after-hours that doesn\u2019t handle tampering, curiosity, or power outages well. Do you have robotics running multiple shifts or other moving equipment that might attract someone\u2019s curiosity? Security cameras would be useful, as would \u201cDo Not Enter\u201d signs. \nSecond, most maintenance staff will notify you (hopefully in writing) about their activities so you can plan accordingly, but what about emergency issues? If they have to fix a leak or a power problem, what are the procedures for safely shutting down systems? Do they have a contact person on your staff in case a problem occurs? Is there hazardous material on-site that requires special handling? Are the maintenance people aware of it and how to handle it? Answers to these questions should be formalized in policy and disseminated to both maintenance and security management personnel, and be made available to new personnel who may not be up to speed.\nIs power shared across other businesses in your building?\nShared power is another significant issue in any building environment. Unless someone paid careful attention to a lab's needs during construction, it can affect any facility. A number of issues can arise from misconfigured or unsupported power systems. Real-life examples of issues a computer support specialist friend of mine has encountered in the past include computers that:\n\n were connected to the same circuit box as heavy duty garage doors. Deliveries would come in early in the morning and when the doors opened the computers crashed.\n were on the same circuit as air conditioners. The computers didn\u2019t crash, but the electrical noise and surging power use created havoc with systems operations and disk drives.\n connected to circuits that didn\u2019t have proper grounding or had separate grounding systems in the same room. Some didn\u2019t have external grounding at all. We worked on a problem with one computer-instrument system that had each device plugged into different power outlets. The computer\u2019s was grounded, but the instrument's power supply wasn\u2019t; once that was fixed everything worked.\n were too close to a radio tower. Every night when the radio system changed its antenna configuration, the computer experienced problems. Today, many devices generate radio signals that might interfere with each other. The fact that they are \u201cdigital\u201d systems doesn\u2019t matter; they are made of analog components.\nIs the power clean, and is the backup generator\u2014if there is one\u2014sufficient to run your systems?\nAnother problem is power continuity and quality. Laboratories depend on clean, reliable power. What will the impact of power outages\u2014lasting anywhere from seconds to days\u2014be on your ability to function? The longer end of the scale is easy; you stop working or relocate critical operations. Generators are one solution option, and we\u2019ll come back to those. The shorter outages, particularly if they are of the power up-down-up variety, are a separate issue. Networkable sensors with alarms and alerts for monitoring power, refrigeration, etc., permitting remote monitoring, may be required. Considerations for these intermittent outages include:\n\n Do you know when they happened? What was their duration? How can you tell? (Again, consider sensor-based monitoring.)\n What effect did intermittent outages have on experiments that were running? Did the systems and instruments reset? Was data lost? Were in-process samples compromised?\n What effect did they have on stored samples? If samples had to be maintained under controlled climate conditions, were they compromised?\n Did power loss and power cycling cause any problems with instrumentation? How do you check?\n Did systems fail into a safe mode?\nHow real are power problems? As Ula Chrobak notes in an August 2020 Popular Science article, infrastructure failures, storms, climate change, etc. are not out of the realm of possibility; if you were in California during that time, you saw the reality first-hand.[10]\nIf laboratory operations depend on reliable power, what steps can we take to ensure that reliability?\nFirst, site selection naturally tops the list. You want to be somewhere that has a reputation for reliable power and rapid repairs if service is lost. A site with buried wiring would be optimal, but that only benefits you a little if the industrial park has buried wiring but is actually fed with overhead wiring. Another consideration is the age of the site: an older established site may have outdated cables that are more likely to fail. The geography is also important. Nearby rivers, lakes, or an ocean might be liable to producing floods, causing water intrusion into wiring. Also, don\u2019t overlook the potential issues associated with earthquakes or nearby industries with hazardous facilities such as chemical plants or refineries. Areas prone to severe weather conditions are an additional consideration.\nSecond, address the overall quality of the building and its infrastructure. This affects buildings you own as well as lease; however, the difference is in your ability to make changes. How old is the wiring? Has it been inspected? Are the grounding systems well implemented? Do you have your own electrical meters, and is your power supply isolated from other units if you are leasing? Will your computers and instruments be on circuits isolated from heavy equipment and garage doors? Make an estimate of your power requirements, then at least double it. Given that, is there sufficient amperage coming into the site to manage all your instruments, computers, HVAC systems, and freezers? How long will you be occupying that space, and is there sufficient power capacity to support potential future expansion?\nThird, consider how to implement generators and battery backup power. These are obvious solutions to power loss, yet they come with their own considerations:\n\n Who has control over generator implementation? If you own the building, you do. If the building is leased, the owner does, and they may not even provide generator back-up power. If not, your best bet\u2014unless you are planning on staying there for a long time\u2014is to go somewhere else; the cost of installing, permitting, and maintaining a generator on a leased site may be prohibitive. A good whole-house system can run up to $10,000, plus the cost of a fueling system. \n How much power will you need and for how long, and is sufficient fuel available? Large propane tanks may need to be buried. Diesel is another option, though fire codes may limit fuel choices in multi-use facilities. The expected duration of an outage is important, also. Often we think perhaps a few hours, but snow, ice, hurricane, tornados, and earthquakes may push that out to a week or more.\n Is the generator\u2019s output suitable for the computers and instruments in your facility? A major problem to acknowledge is electrical noise: too much and you\u2019ll create more problems than you would have if the equipment had just been shut down.\n What is the startup delay of the generator? A generator can take anywhere from a few seconds to several minutes to get up to speed and produce power. Can you afford that delay? Probably not.\nThe answer to the problems noted in the last two bullets is battery backup power. These can range from individual units that are used one-per-device, like home battery backups for computers and other equipment, to battery-walls that are being offered for larger applications. The advantage is that they can come online anywhere from instantly (i.e., always-on, online systems) to a few milliseconds for standby systems. The always-on, online options contain batteries that are constantly being charged and at the same time constantly providing power to whatever they are connected to. More expensive than standby systems, they provide clean power even from a source that might otherwise be problematic. On the other hand, standby systems are constantly charging but pass through power without conditioning; noisy power in, noisy power out until a power failure occurs.\n\nSecurity and the working environment \nWhen we are looking at security as a topic, we have keep in mind that we are affecting people's ability to work. Some of the laboratory's work is done at the lab bench or on instruments (which, depending on the field you\u2019re in, could range from pH meters to telescopes). However, significant work occurs away from bench, thinking and planning wherever a thought strikes. What kind of flexibility do you want people to have? Security will often be perceived as stuff that gets in the way of personnel's ability to work, despite the fact that well-implemented security protects their work. \nWe need to view security as a support structure enabling flexibility in how people work, not simply as a series of barriers that frustrate people. You can begin by defining the work structure as you\u2019d like it to be, at the same time recognizing that there are two sides to lab work: the intellectual (planning, thinking, evaluating, etc.) and the performed (where you have to work with actual samples, equipment, and instruments). One can be done anywhere, while the other is performed in a specific space. The availability of computers and networks can blur the distinction. \nKeeping these things in mind, any security planning should consider the following:\n\n How much flexibility do personnel want in their work environment vs. what they can actually have? In some areas of work, there may be significant need for lockdown with little room for flexibility, while other areas may be pretty loose.\n Do people want to work from remote places? Does the nature of the work permit it? This can be motivated by anything from \u201cthe babysitter is sick\u201d to \u201cI just need to get away and think.\u201d\n While working remotely, do people need access to lab computers for data, files (i.e., upload\/download), or to interact with experiments? Some of this can be a matter of respecting people\u2019s time. If you have an experiment running overnight or during the weekend, it would be nice to check the status remotely instead of driving back to work.\n Do people need after-hours access to the lab facilities?\nThe answers to these planning questions lay the groundwork for hardware, software, and security system requirements. Can you support the needs of personnel, and if so, how is security implemented to make it work? Will you be using gateway systems to the lab network, with additional logins for each system, two-factor authentication, or other mechanisms? The goal is to allow people to be as productive as possible while protecting the organization's resources and meeting regulatory requirements. That said, keep in mind that unless physical and virtual access points are well controlled, others may compromise the integrity of your facility and its holdings.\nEmployees need to be well educated in security requirements in general and how they are implemented in your facility. They need to be a willing part of the processes and not grudgingly accepting them; that lack of willingness to work within the system is a security weak point, things people will try to circumvent. One obvious problem is with username-password combinations for computer access; rather than typing that information in, biometric features are faster and less error prone.\nThat said, personnel should readily accept that no system should be open to unauthorized access, and that hierarchical levels of control may be needed, depending on the type of system; some people will have access to some capabilities and not others. This type of \"role-based\" access shouldn\u2019t be viewed as a matter of trust, but rather as a matter of protection. Unless the company is tiny, senior management, for example, shouldn\u2019t have administrative system level access to database systems or robotics. If management is going to have access to those levels, ensure they know exactly what they are doing. By denying access to areas not needed in a role-based manner, you limit the ability of personnel to improperly interrogate or compromise those systems for nefarious purposes.\nWhat are your security control requirements?\nFigure 16 lists some of the key areas of concern for security controls. Some we\u2019ll touch on, others we'll leave to those better informed (e.g., see Tulsi 2019[11] and Riley 2020[12]).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 16. Key areas of concern for security controls\n\n\n\nWhat is your policy on system backups?\nWhen it comes to your computer systems, are you backing up their K\/I\/D? If so, how often? How much K\/I\/D can you afford to lose? Look at your backups on at least three levels. First, backing up the hard drive of a computer protects against failure of that computer and drive. Backing up all of a lab\u2019s data systems to an on-site server in a separate building (or virtualized locally) protects against physical damage to the lab (e.g., fire, storms, earthquake, floods, etc.). Backing up all of a lab\u2019s data systems to a remote server (or virtualized remotely) provides even more protection against physical damage to the lab facility, particularly if the server is located someplace that won\u2019t be affected by the same problems your site may be facing. It should also be somewhere that doesn\u2019t compromise legal control over your stuff; if it is on a third-party server farm in another country, that country\u2019s laws apply to access and seizure of your files if legal action is taken against you.\nShould your various laboratory spaces and components be internet-connected?\nWhen looking at your lab bench spaces, instruments, database systems, etc., determine whether they should be connected to the internet. This largely depends on what capabilities you expect to gain from internet access. Downloading updates, performing online troubleshooting with vendors, and conducting online database searches (e.g., spectra, images, etc.) are a few useful capabilities, but are they worth the potential risk of intrusion? Does your IT group have sufficient protection in place to allow access and still be protected? Note, however, any requirement for a cloud-based system would render this a moot point.\nLab systems should be protected against any intrusion, including vendors. Vendor-provided files can be downloaded to flash drives, which can then be checked for malware and integrity before being manually transferred to lab systems. Consider what is more important: convenience or data protection? This may give you more to think about when you consider your style of working (e.g., remote access). However, having trusted employees access the lab network is different than third-parties.\n\nSummary \nWe\u2019ve only been able to touch on a few topics; a more thorough review would require a well-maintained document, as things are changing that quickly.[k] In many labs, security is a layered activity, where as the work of the lab is planned, security issues are then considered. We\u2019d be far better off if security planning was instead conducted in concert with lab systems planning; support for security could would then become part of the product selection criteria.\n\nSixth goal: Acquiring and developing \"products\" that support regulatory requirements \nProducts should be supportable. That seems pretty simple, but what exactly does that mean? How do we acquire them, and more importantly, how do we develop them? The methods and procedures you develop for lab use are \u201cproducts\u201d\u2014we\u2019ll come back to that.\nFirst, here\u2019s an analogy using an automobile. The oil pan on a car may need to be replaced if it is leaking due to damage or a failed gasket; if it isn\u2019t repaired, the oil can leak out. Some vehicles are more difficult to work on than others given their engineering. For example, replacing the oil pan in some cars requires you to lift the engine block out of the car. That same car design could also force you to move the air conditioner compressor to change spark plugs. In the end, some automobile manufactures have built a reputation for cars that are easy to service and maintain, which translates into lower repair costs and longer service life. \nHow does that analogy translate to the commercial products you purchase for lab use, as well as the processes and procedures you develop for your lab? The ability to effectively (i.e., with ease, a low cost, etc.) support a product has to be baked into the design from the start. It can\u2019t be retrofitted.\nLet\u2019s begin with the commercial products you purchase for lab use, including instruments, computer systems, and so on. One of the purchase criteria for those items is how well they are supported: mature products should have a better support infrastructure, built up over time and use. However, that doesn\u2019t always translate to high-quality support; you may find a new product getting eager support because the vendor is heavily invested in market acceptance, working the bugs out, and using referral sites to support their sales. When it comes to supporting these commercial products, we expect to see:\n\n User Guides \u2013 This should tell you how the device works, what the components are (including those you shouldn\u2019t touch), how to use the control functions, what the expected operating environment is, what you need to provide to make the item usable, and so on. For electronic devices with control signal communications and data communications, the vendor will describe how it works and how they expect it to be used, but not necessarily how to use it with third-party equipment. There are limitations of liability and implied support commitments that they prefer not to get involved with. They provide a level of capability, while it\u2019s largely left up to you to make it work in your application.\n Training materials \u2013 This will take you from opening the box, setting up whatever you\u2019ve purchased, and walking through all the features and some examples of their use. The intent is to get you oriented and familiar with using it, with the finer details located in user guides. Either this document or the user guide should tell you how to ensure that the device is installed and operating properly, and what to do if it isn\u2019t. This category can also include in-person short courses as well as online courses (an increasingly popular option as something you can do at your convenience).\n Maintenance and troubleshooting manuals \u2013 This material describes what needs to be periodically maintained (e.g, installing software upgrades, cleaning equipment, etc.) and what to do if something isn\u2019t working properly.\n Support avenues - Be it telephone, e-mail, or online chat, there are typically many different ways of reaching the vendor for help. Online support can also include a \u201cknowledgebase\u201d of articles on related topics, as well as chat functions.\n User groups \u2013 Whether conducted in-person or online, venues for giving users a chance to solve problems and present material together can also prove valuable.\nFrom the commercial side of laboratory equipment and systems, support is an easy thing to deal with. If you have good products and support, people will buy them. If your support is lacking, they will go somewhere else, or you will have fostered the development of a third-party support business if your product is otherwise desirable.\nFrom the system user\u2019s perspective, lab equipment support is a key concern. Users typically don\u2019t want to take on a support role in the lab as that isn\u2019t their job. This brings us to an interesting consideration: product life cycles. You buy something, put it to use, and eventually it has to be upgraded (particularly if it involves software) or possibly replaced (as with software, equipment, instruments, etc.). Depending on how that item was integrated into the lab\u2019s processes, this can be a painful experience or an easy one. Product life cycles are covered in more detail later in this section, but for now know they are important because they apply, asynchronously, to every software system and device in your lab. Upgrade requirements may not be driven by a change in the functionality that is important to the lab, but rather due to a change to an underlying component, e.g., the computer's operating system. The reason that this is important in a discussion about support is this: when you evaluate a vendor's support capabilities, you need to cover this facet of the work. How well do they evaluate changes in the operating system (OS) in relation to the functionality of their product? Can they advise you about which upgrades are critical and those that can be done at a more convenient time? If a change to OS or a database product occurs, how quickly do they respond?\nNow that we have an idea what support means for commercial products, let\u2019s consider what support means for the \"products\"\u2014i.e., the procedures and methods\u2014developed in your lab.\nThe end result of a typical laboratory-developed method is a product that incorporates a process (Figure 17). This idea is nothing new in the commercial space. Fluid Management Systems, Inc. has complex sample preparation processing systems as products[13], as do instrument vendors that combine autosamplers, an instrument, and a data system (e.g., some of Agilent\u2019s PAL autosampler systems incorporate sample preparation processing as part of their design[14]). Those lab methods and procedures can range from a few steps to an extensive process whose implementations can include fully manual execution steps, semi-automated steps (e.g., manual plus instrumentation), and fully automated steps. In the first two cases, execution can occur with either printed or electronic documentation, or it can be managed by a LES. However, all of these implementations are subject to regulatory requirements (commercial products are subject to ISO 9000 requirements).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 17. A laboratory process\n\n\n\nThe importance of documentation \nRegulatory requirements and guidelines (e.g., from the FDA, EPA, ISO, etc.) have been with production and R&D for decades. However, some still occasionally question those regulations' and guidelines' application to research work. Rather than viewing them as hurdles which a lab must cross to be deemed qualified, they should be viewed as hallmarks of a well-run lab. With that perspective, they remain applicable for any laboratory.\nFor purposes of this guide, there is one aspect of regulatory requirements that will be emphasized here: process validation, or more specifically the end result, which is a validated process. Laboratory processes, all of which have to be validated, are essentially products for a limited set of customers; in many cases its one customer, in others the same process may be replicated in other labs as-is. The more complex the implementation, and the longer the process is expected to be in use, the more important it is to incorporate some of the tools from commercial developers into lab development (Table 1). However, regardless of development path, complete documentation is of the utmost concern. \n\n\n\n\n\n\n\nTable 1. Process and product development documentation\n\n\nLaboratory-developed products\n\nCommercially developed products\n\n\n\u2022 user requirements document\r\n\u2022 functional specification\r\n\u2022 system design and testing\r\n\u2022 implementation\r\n\u2022 installation, operational, and performance qualification\r\n(IQ, OQ, PQ) evaluation and acceptance\n\n\u2022 product requirements document\r\n\u2022 engineering or functional specification\r\n\u2022 product testing protocol\r\n\u2022 product readiness for market acceptance criteria\r\n \r\nPLUS product support elements like:\r\n\u2022 user guides\r\n\u2022 training materials\r\n\u2022 maintenance and troubleshooting guides\r\n\u2022 support mechanisms\r\n\u2022 user groups\n\n\n\nDocumentation is valuable because:\n\n It's educational: Quality documentation ensures those carrying out the process or maintaining it are thoroughly educated. Written documentation (with edits and audit trails, as appropriate) acts as a stable reference point for how things should be done. The \u201cfollow-me-and-I\u2019ll-show-you\u201d approach is flawed. That method depends on someone accurately remembering and explaining the details while having the time to actually do it, all while hoping bad habits don't creep in and become part of \u201chow it\u2019s done.\u201d\n It informs: Quality documentation that is accessible provides a reference for questions and problems as they occur. The depth of that documentation, however, should be based on the nature of the process. Even manual methods that are relatively simple need some basic elements. To be innformative, it should address numerous questions. Has the instrument calibration been accurately verified? How do you tell, and how do you correct the problem if the instrument is out of calibration? What information is provided about reagents, including their age, composition, strength, and purity? When is a technician qualified to use a reagent? How are reference materials incorporated as part of the process to ensure that it is being executed properly and consistently?\nNote that the support documents noted in Table 1 are not usually part of process validation. The intent of process validation is to show that something works as expected once it is installed.\nOne aspect that hasn\u2019t been mentioned so far is how to address necessary change within processes. Any lab process is going to change over time. There may be a need for increased throughput, lower operating costs, less manual work, the ability to run over multiple shifts, etc. There may also be new technologies that improve lab operations that eventually need to get incorporated into the process. As such, planning and process documentation should describe how processes are reviewed and modified, along with any associated documentation and training. This requires the original project development to be thoroughly documented, from functionality scoping to design and implementation. By including process review and modification as part of a process allows that process to be upgraded without having to rebuild everything from scratch. This level of documentation is rarely included due to the initial cost and impact on the schedule. It will affect both, but it will also show its value once changes have to be made. In the end, by adding process review and modification mechanism, you ensure a process is supportable.\nTo be clear, the initial design and planning of process and methods has to be done well for a supportable product. This means keeping in mind future process review and modification even as the initial process or method is being developed. It\u2019s the difference between engineering a functional and supportable system and \u201cjust making something that works.\u201d Here are three examples:\n\n One instrument system vendor, in a discussion between sessions of a meeting, described how several of his customers successfully connected a chromatography data system (CDS) to a LIMS. It was a successful endeavor until one of the systems had to be upgraded, then everything broke. The programmer had made programming changes to areas of the packages that they shouldn\u2019t have. When the upgrade occurred, those changes were overwritten. The project had to be scrapped and re-developed.\n A sample preparation robotics system was similarly implemented by creating communications between devices in ways that were less than ideal. When it came time for an upgrade to one device, the whole system failed.\n A consultant was called in to evaluate a project to interface a tensile tester to a computer, as the original developer had left the company. The consultant recommended the project be scrapped and begun anew. The original developer had not left any design documentation, the code wasn\u2019t documented, and no one knew if any of it worked or how it was supposed to work. Trying to understand someone else\u2019s programming without documentation assistance is really a matter of trying to figure out their thinking process, and that can be very difficult.\nThere are a number of reasons why problems like this exist. Examples include lack of understanding of manual and automated systems design and engineering methodologies and pressure from management (e.g., \u201chow fast can you get it done,\u201d \u201ckeep costs down,\u201d and \u201cwe\u2019ll fix it in the next version\u201d). Succumbing to these short-term views will inevitably come back to haunt you in the long-term. Upgrades, things you didn\u2019t think of when the original project was planned, and support problems all tend to highlight work that could have been done better. Another saying that frequently comes up is \u201cthere is never time to do it right, but there is always time to do it over,\u201d usually at a considerably higher cost.\n\nAdditional considerations in creating supportable systems and processes \nSingle-vendor or multi-vendor?\nLet\u2019s assume we are starting with a manual method that works and has been fully validated with all appropriate documentation. And then someone wants to change that method in order to meet a company goal such as increasing productivity or lowering operational costs. Achieving goals like those usually means introducing some sort of automation, anything from automated pipettes to instrumentation depending on the nature of the work. Even more important, if a change in the fundamental science underlying the methodology is proposed, that would also require re-validation of the process.\nJust to keep things simple, let\u2019s say the manual process has an instrument in it, and you want to add an autosampler to keep the instrument fed with samples to process. This means you also need something to capture and process the output or any productivity gains on the input may be lost in handling the output. We\u2019ll avoid that discussion because our concern here is supportability. There are a couple directions you can go in choosing an add-on sampler: buy one from the same vendor as the instrument, or buy it from another vendor because it is less expensive or has some interesting features (though unless those features are critical to improving the method, they should be considered \u201cnice but not necessary\u201d).\nHow difficult is it going to be to make any physical and electronic (control) connections to the autosampler? Granted, particularly for devices like autosamplers, vendors strive for compatibility, but there may be special features that need attention. You need to consider not just the immediate situation but also how things might develop in the future. If you purchase the autosampler from the same vendor as the instrument and control system, they are going to ensure that things continue to work properly if upgrades occur or new generations of equipment are produced (see the product life cycle discussion in the next section). If the two devices are from different vendors, compatibility across upgrades is your issue to resolve. Both vendors will do what they can to make sure their products are operating properly and answer questions about how they function, but making them work together is still your responsibility.\nFrom the standpoint of supportability, the simpler approach is the easiest to support. Single-vendor solutions put the support burden on them. If you use multi-vendor implementations, then all the steps in making the project work have to be thoroughly documented from the statement of need, through to the functional specifications and the finished product. The documentation may not be very long, but any assistance you can give someone who has to work with the system in the future\u2014including yourself (i.e., \u201cwhat was I thinking when I did this?\u201d)\u2014will be greatly appreciated.\nOn-device programming or supervisory control?\nAnother consideration is for semi- or fully-automated systems where a component is being added or substituted. When we are looking at programmable devices, one approach is to make connections between devices via on-device programming. For example, say Device A needs to work with Device B, so programming changes are made to both to accomplish the task. While this can be made to work and be fully documented, it isn\u2019t a good choice since changing one of them (via upgrade or swap) will likely mean the programming has to be re-implemented. A better approach is to use a supervisory control system to control them both, and others that may be part of the same process. It allows for a more robust design, easier adaptations, and smoother implementation. It should be easier to support since programming changes will be limited to communications codes.\nThird-party developers and contractors?\nFrequently, third parties are brought in to provide services that aren\u2019t available through the lab or onsite IT staff. For example, a functional specification usually describes what you want as the end result and what their product is supposed to do, not how it is supposed to do it. This is left to the developer to figure out. You need to add supportability as a requirement, that the end result not only meet regulatory requirements, but that it also is designed and documented with sufficient information to have someone unfamiliar with the project understand what would have to be done if a change were made, which also requires you to think about where changes might be made in the future. This includes considering what components might be swapped for newer technologies, handling software upgrades (and what might break as a result of them), and knowing what to do if components reach their supported end-of-life and have to be replaced.\nConsulting firms may respond with \u201cif something needs to be changed or fixed, just call us, we built it,\u201d which sounds reasonable. However, suppose the people who built it aren\u2019t there anymore or aren't available because they're working on other projects. The reality is the \u201ccompany\u201d didn\u2019t build the product, people working for them did.\n\nProduct life cycles \nWhen discussing product life cycles, whether it's digital products or hardware systems, the bottom line problem is this: what do you do when a critical product needs to be updated or replaced? This can be an easy issue or a very painful one, depending on how much thought went into the design of the original procedure using that device. It's generally easy if you had the forethought of noting \u201csomeday this is going to be replaced, so how do we simplify that?\" It's more difficult if you, through wiring or software, linked devices and systems together and then can\u2019t easily separate them, particularly if no one documented how that might be accomplished. It\u2019s all a matter of systems engineering done well.\nNote: This material was originally published in Computerized Systems in the Modern Laboratory: A Practical Guide.\nAn analog pH meter\u2014as long as it has been maintained\u2014will still work today. So will double-pan balances, again as long as they have been maintained and people are proficient in their use. Old lab equipment that will still function properly has been replaced with more modern equipment due to better accuracy, ease of use, and other factors. Analog instruments can still be found operating decades after their design end-of-life. It is in the digital realm that equipment that should work (and probably does) but can\u2019t be used after a few years of service. The rate of technology change is such that tools become obsolete on the order of a half-decade. For example, rotating disks, an evolving computer staple that replaced magnetic tape drives, are now being replaced with solid-state storage.\nDigital systems require two components to work: hardware (e.g., the computer, plus interfaces for the hard disk, and ports for cable connections) and software (e.g., operating systems plus software drivers to access the hardware). Both hardware packaging and operating systems are changing at an increasing rate. Hardware systems are faster, with more storage, and operating systems are becoming more complex to meet consumer demands, with a trend toward more emphasis on mobile or social computing. Those changes mean that device interfaces we rely on may not be there in the next computer you have to purchase. The RS-232 serial port, a standard for instrument connections, is being replaced with USB, Firewire, and Thunderbolt connections that give support to a much wider range of devices and simplify computer design, with more usable and less costly devices. It also means that the instrument with the RS-232 interface may not work with a new computer due to there being no RS-232 ports, and the operating system may also no longer be compatible with the instrumentation.\nOne aspect of technology planning in laboratory work is change management, specifically the impact of technology changes and product life cycles on your ability to work. The importance of planning around product life cycles has taken on an added dimension in the digital laboratory. Prior to the use of computers, instruments were the tools used to obtain data, which was recorded in paper laboratory notebooks. End result: getting the data and recording and managing it were separate steps in lab work. If the tools were updated or replaced, the data recorded wasn\u2019t affected. In the digital realm, changes in tools can affect your ability to work with new and old data and information. The digital-enabled laboratory requires planning, with a time horizon of decades to meet legal and regulatory requirements. The systems and other tools you use may not last for decades; in fact, they will probably change several times. However, you will have to plan for the transfer of the data and information they contain and address the issue of database access and file formats. The primary situation to avoid is having data in files that you can\u2019t read.\nWhile we are going to begin looking at planning strategies for isolated products as a starting point, please keep in mind that in reality products do not exist in isolation. The laboratory\u2019s K\/I\/D is increasingly interconnected, and changes in one part of your overall technology plan can have implications across your lab's working technology landscape. The drive toward integration and paperless laboratories has consequences that we are not fully prepared to address. We\u2019ll start with the simpler cases and build upon that foundation.\nDigital products change for a number of reasons: \n\n The underlying software that support informatics applications could change (e.g., operating systems, database systems), and the layers of software that build on that base have to be updated to work properly.\n Products could see improvements due to market research, customer comments, and competitive pressures.\n Vendors could get acquired, merge with another company, or split up, resulting in products merging or one product being discarded in favor of another.\n Your company could be acquired, merge with another company, or split into two or more organizations.\n Products could simply fail.\n Your lab could require a change, replacing older technologies with systems that provide more capability.\nIn each of these cases, you have a decision to make about how K\/I\/D is going to be managed and integrated with the new system(s).\nBut how often do digital products change? Unfortunately, there isn't much detailed information published about changes in vendor products. Historically, operating systems used to be updated with new versions on an annual basis, with updates (e.g., bug fixes, minor changes) occurring more frequently. With a shift toward subscription services, version changes can occur more frequently. The impact of an OS version change will vary depending on the OS. Some vendors take responsibility and control for the hardware and software, and as a result, upgrades support both the hardware and OS until the vendor no longer supports new OS\nversions on older systems. Other computer systems, where the hardware and software components come from different vendors, can result in the inability to access hardware components due to an upgrade. The OS upgrade only supports certain hardware features. Support for specific add-on equipment (including components provided by the computer vendor) may require finding and reinstalling drivers from the original component vendor. As for the applications that run on operating systems, they will need to be tested with each OS version change. \nApplications tend to be updated on an irregular basis, for both direct installs and for cloud-hosted solutions. Microsoft Office and Adobe\u2019s Creative Cloud products may be updated as they see a need. Since both product suites are now accessed via the internet on a subscription basis (software as a service or SaaS), user action isn\u2019t required. Lab-specific applications may be upgraded or updated as the vendor sees a need; SaaS implementations are managed by the vendor according to the vendor's internal planning. Larger, stable vendors may provide upgrades on a regular, annual basis for on-site installations. Small vendors may only update when a significant change is made, which might include new features, or when forced to because of OS changes. If those OS compatible changes aren\u2019t made, you will see yourself running software that is increasingly out-of-date. That doesn\u2019t necessarily mean it will stop working (for example, Microsoft dropped support for Windows XP in the Spring of 2014, and computers running it didn\u2019t suddenly stop). What it does mean is that if your computer hardware has to be replaced, you may not be able to re-install a working copy of the software. The working lifetime of an application, particularly a large one, can be on the order of a decade or more. Small applications depend upon market acceptance and the vendor\u2019s ability to stay in business. Your need for access to data may exceed the product's life.\nThe perception of the typical product life cycle runs like this: a need is perceived; product requirements are drafted; the product is developed, tested, and sold, based on market response to the product; new product requirements are determined; and the cycle continues. The reality is a bit more complicated. Figure 18 shows a more realistic view of a product\u2019s life cycle. The letters in the circles refer to key points where decisions can have an impact on your lab (\"H\" = high, \"M\" = medium, \"L\" = low).\n\n\n\n\n\n\n\n\n\n Figure 18. The product life cycle\n\n\n\nThe process begins with an initial product concept, followed by the product's development, introduction, and marketing programs, and finally its release to customers. If the product is successful, the vendor gathers customer comments, analyzes competitive technologies and any new technologies that might be relevant, and determines a need for an upgrade.\nThis brings us to the first decision point: is an upgrade possible with the existing product? If it is, the upgrade requirements are researched and documented and the process moves to development, with generally a low impact on users. \u201cGenerally\u201d because it depends on the nature of the product and what modifications, changes, and customizations have been made by the user. If it is an application that brings a data file in, processes it, and then saves the result in an easily accessible file format, allowing no user modifications to the application itself, \u201clow impact\u201d is a fair assessment. Statistical analysis packages, image processing, and others such applications fall into this set. Problems can arise when user modifications are overwritten by the upgrade and have to be reinstalled (only a minor issue if it is a plug-in with no programming changes) or re-implemented by making programming changes (a major problem since it requires re-validation). Normally any customization (e.g., naming database elements) and data held within an application's database should be transferred without any problems, though you do need to make some checks and evaluations to ensure that this is the case. This is the inner loop of Figure 18.\nSignificant problems can begin if the vendor determines that the current product generation needs to be replaced to meet market demands. If this is a hardware product (e.g., pH meter, balance, instrument, computer, etc.) there shouldn\u2019t be any immediate impact (the hardware will continue to work). However, once there is a need for equipment replacement, it becomes a different matter; we\u2019ll pick this thread up later when we discuss product retirement.\nSoftware is a different situation. The new generation of software may not be compatible with the hardware you currently have. What you have will still work, but if there are features in the new generation that you\u2019d like to have, you may find yourself having to re-implement any software changes that you\u2019ve made to the existing system. It will be like starting over again, unless the vendor takes pains to ensure that the upgrade installation is compatible with the existing software system. This includes assurances that all the data, user programming, settings, and all the details you\u2019ve implemented to make a general product work in your environment can successfully migrate. You will also have to address user education, and plan for a transition from the old system to the new one.\nOne problem that often occurs with new generations of software is change in the underlying data file structures. The vendor may have determined that in order to make the next generation work and to be able to offer the features they want included, the file structure and storage formats will change. This will require you to re-map the existing file structure into the new one. You may also find that some features do not work the same as they did before and your processes have to be modified. In the past, even new versions of Microsoft Office products have had compatibility issues with older versions. In large applications such as informatics or instrument data systems (e.g., multi-user chromatography data system), changes in formats can be significant. It can have an effect on importing instrument data into informatics products. For example, some vendors and users use a formatted PDF file as a means of exchanging instrument data with a LES, SDMS, or ELN. If the new version of an instrument data system changes its report formatting, the PDF parsing routine will have be updated.\nAt this point, it's important to note that just because a vendor comes out with a new software or hardware package doesn\u2019t mean that you have to upgrade. If what you have is working and the new version or generation doesn\u2019t offer anything of significant value (particularly when the cost of upgrading and the impact on lab operations is factored in) then bypass the upgrade. Among the factors that can tug you into an upgrade is the potential loss of support for the products you are concerned with.\nWhat we\u2019ve been discussing in the last few paragraphs covers the outer loop to the right of Figure 18. The next point we need to note in that figure is the \u201cNo\u201d branch from \u201cNew Product Generation Justified?\u201d and \u201cProduct Fails,\u201d both of which lead to product retirement. For both hardware and software, you face the loss of customer support and the eventual need for product replacement. In both cases, there are steps that you can take to manage the potential risks.\nTo begin with, unless the vendor is going out of business, they are going to want to maintain a good relationship with you. You are a current and potential future customer, and they\u2019d like to avoid bad press and problems with customer relationships. Explain how the product retirement is going to affect you and get them to work with you on managing the issue; you aren\u2019t the only one affected by this (see the commentary on user groups later). If you are successful, they will see a potential liability turn into a potential asset: you can now be a referral for the quality of their customer service and support. Realistically, however, your management of product retirement or major product changes has to occur much earlier in the process.\nYour involvement begins at the time of purchase. At that point you should be asking what the vendors update and upgrade policies are, how frequently they occur, what the associated costs are, how much advanced notice they give for planning, and what level of support is provided. In addition, determine where the product you are considering lies in the product\u2019s life cycle. Ask questions such as:\n\n Is it new and potentially at-risk for retirement due to a lack of market acceptance? If it is and the vendor is looking for reference sites, use that to drive a better purchase agreement. Make sure that the product is worth the risk, and be prepared in case the worst-case scenario occurs.\n Is it near the end-of-life with the potential for retirement? Look at the frequency of updates and upgrades. Are they tailing off or is the product undergoing active development?\n What is the firm\u2019s financial position? Is it running into declining sales or are customers actively seeking it? Is there talk of acquisitions or mergers, either of which can put the product's future into question?\nYou should also ask for detailed technical documents that describe where programming modifications are permitted and preserved against vendor changes, and how data will be protected, along with any tools for data migration. Once you know what the limitations are for coding changes, device additions, and so on are, the consequences of deviating from them are your responsibility; whatever you do should be done deliberately and with full awareness of their impact in the future.\nOne point that should be clarified during the purchase process is whether you are purchasing a product or a product license. If you a purchasing a product, you own it and can do what you like with it, at least for hardware products. Products that are combinations of hardware and software may be handled differently since the hardware won\u2019t function without the software. Licenses are \u201crights to use\u201d with benefits and restrictions. Those should be clearly understood, as well as what you can expect in terms of support, upgrades, the ability to transfer products, how products can be used, etc. If there are any questions, the time to get them answered is before you sign purchase agreements. You have the best leverage for gaining information and getting reasonable concessions that are important to you while the vendor is trying to sell you something. If you license a product, the intellectual property within the product belongs to the vendor while you own your K\/I\/D; if you decide to stop using a product, you should have the ability to extract your K\/I\/D in a usable form.\nAnother point: if the product is ever retired, what considerations are provided to you? For a large product, they may not be willing to offer documented copies of the code so that you can provide self-support, but a small company trying to break into the market might. It doesn\u2019t hurt to ask and get any responses in writing, don\u2019t trust someone\u2019s verbal comments; they may not be there when upgrades or product retirement occurs. Additionally, it's always beneficial to conduct negotiations on purchase and licenses in cooperation with your company's IT and legal groups. IT can advise on industry practices, and the legal department\u2019s support will be needed for any agreements.\nAnother direction you should take is participating in user groups. Most major vendor and products have user groups that may exist as virtual organizations on LinkedIn, Yahoo, or other forums. Additionally, they often have user group meetings at major conferences. Company-sponsored group meetings provide a means for learning about product directions, raising issues, discussing problems, etc. Normally these meeting are divided into private (registered users only) and public sessions, the former being the most interesting since they provide a means of unrestricted comments. If a new version or upgrade is being considered, it will be announced and discussed at group meetings. These will also provide a mechanism for making needs known and if a product is being retired, lobbying for support. The membership contact list will provide a resource for exchanging support dialogue, particularly if the vendor is reluctant to address points that are important to you.\nIf a group doesn\u2019t exist, start a virtual conference and see where it goes. If participation is active, let the vendor know about it; they may take an interest and participate or make it a corporate function. It is in a company's best interest to work with its customers rather than antagonize them. Your company\u2019s support may be needed for involvement in, or starting, user groups because of the potential for liability, intellectual property protection, or other issues. Activities performed in these types of groups can be wide-ranging, from providing support (e.g., trading advice, code, tutorials, etc.) and sharing information (e.g., where to get parts, information for out-of-warranty products) to identifying and critiquing repair options and meeting informally for conferences.\nThe key issue is to preserve your ability to carry out your work with as little disruption as possible. That means you have to protect your access to the K\/I\/D you\u2019ve collected, along with the ability to work with it. In this regards, software systems have one possible advantage: virtualization.\nVirtualization: An alternative to traditional computing models\nThere are situations in laboratory computing that are similar to the old joke \u201cyour teeth are in great shape but the gums have to go.\u201d The equivalent situation is running a software package and finding out that the computer hardware is failing and the software isn\u2019t compatible with new equipment. That can happen if the new computer uses a different processor than the one you are working with. An answer to the problem is a technology called virtualization. In the context of the joke, it lets you move your teeth to a new set of gums; or to put it another way, it allows you to run older software packages on new hardware and avoid losing access to older data (there are some limitations).\nBriefly put, virtualization allows you to run software (including the operating system) designed for one computer on an entirely different system. An example: the Windows XP operating system and applications running on a Macintosh computer using the MAC OS X operating system via VMware\u2019s Fusion product. In addition to rescuing old software, virtualization can:\n\n reduce computing costs by consolidating multiple software packages on servers;\n reduce software support issues by preventing operating system upgrades from conflicting with lab software;\n provide design options for multiple labs using informatics products without incurring hardware costs and giving up lab space to on-site computers; and\n reduce interference between software packages running on the same computer.\nRegarding that last benefit, it's worth noting that with virtualization, adding software packages means each gets its own \u201ccomputer\u201d without additional hardware costs. Product warrantees may state the software warrantee is limited to instances where the software is installed on a \u201cclean machine\u201d (just the current operation system and that software package, nothing else). Most people put more than one application on a computer, technically voiding the warrantee. Virtualized containers let you go back to that clean machine concept without buying extra hardware.\nIn order to understand virtualization, we have to discuss computing, but just the basics. Figure 19 shows an arrangement of the elements. When the computer is first turned on, there are three key items engaged: the central processing unit (CPU), memory, and mass storage. The first thing that happens (after the hardware startup sequence) is that portions of the operating system are placed in the memory where the CPU can read instructions and begin working. The key point is that the operating system, applications, and files are a collection of binary data elements (words) that are passed on to the CPU.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 19. Key elements in a computer system\n\n\n\nThe behavior of the CPU can be emulated by a software program. We can have a program that acts like an Intel processor for example, or a processor from another vendor. If we feed that program the instructions from an application, it will execute that application. There are emulators for example, that will allow your computer to emulate an Atari 2600 game console and run Asteroids. There are also emulators for other game consoles, so your computer can behave like any game console you like, as long as you have an emulator for it. Each emulator has all the programming needed to execute copies of the original game programming. They don\u2019t wear out or break. This configuration is shown in Figure 20.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 20. Emulation on a computer\n\n\n\nYou have a series of game emulators, each with its collection of games. Any game emulator can be loaded into memory and execute a game from its collection; games for other emulators won\u2019t work. Each game emulator and game collection is called a container. When you want to play, you access the appropriate container and go. If the mass storage is on a shared server, other people can access the same containers and run the games on their computers without interfering with each other.\nHow does this apply to a computer with failing hardware that is running a data analysis program? Virtualized systems allow you to make a copy of mass storage on the computer, create a container containing the CPU emulator, OS, applications, and data files, and place it on a server for later access. The hardware no longer matters because it is being replaced with the CPU emulator. Your program\u2019s container can be copied, stored, backed up, etc. It will never wear out or grow old. When you want to run it, access the container and run it. A server can support many containers.\nThere are some restrictions, however. First, most computers that are purchased from stores or online come pre-loaded with an operating system. Those operating systems are OEM (original equipment manufacturer) copies of the OS whose license cost is buried in the purchase price. They can\u2019t be copied or transferred legally, and some virtual servers will recognize OEM copies and not transfer them. As a result, in order to make a virtualized container, you need a fully licensed copy of the OS. Your IT group may have corporate licenses for widely used operating systems, so that may not pose a problem. Next, recognize that some applications will require a separate license for use on a virtualized system. As frequently noted, planning ahead is key: explore this option as part of the purchase agreement: you may get a better deal. Third, it's important to note that virtualized systems cannot support real-time applications such as direct analog, clock-driven, time-critical data acquisition from an instrument. The virtualized software shares resources with other containers in a time-sharing mode, and as a result the close coordination for data acquisition will not work. Fortunately, direct data acquisition (as contrasted with computer-to-instrument communications via RS-232, USB, Ethernet, etc.) is occurring less often in favor of buffered data communications with dedicated data acquisition controllers, so this is becoming less of a problem. If you need direct computer-controlled data acquisition and experiment control, this isn\u2019t the technology for you. Finally, containerized software running on virtualized systems cannot access hardware that wasn\u2019t part of the original configuration. If the computer you are using has a piece of hardware that you\u2019d like to use but wasn\u2019t on the original virtualized computer, it won\u2019t be able to use it since it doesn\u2019t have the software drivers to access it.\nIf the applications software permits it, applications can have shared access to common database software. A virtualized LIMS may be a good way to implement the application since it doesn\u2019t require hardware in the lab and uses servers that are probably under IT control, and as a result the systems are backed up regularly. The major hang-up on these installations is instrument connections. IT groups tend to get very conservative about that subject. Middleware could help isolate actual instrument connections from the network and could potentially resolve the situation. The issue is part technical, part political. However, virtualized LIMS containers still prove beneficial for educational purposes. A student can work with the contents of a container, experiment as needed, and when done dismiss the container without saving it; the results of the experiments are gone, mistakes and all.\nThere are different types of virtualization. One has containers sharing a common emulator and operating system. As a result, you update or upgrade the emulator software and\/or operating system once and the change is made across all containers. That can cause problems for some applications; however, they can be moved to a second type of virtualization in which each container has it own copy of the operating system and they can be excluded from updates.\nIf you find this technology appealing, check with your vendor to see if the products of interest will function in a virtualized environment (not all will). Carefully ask questions, perhaps asking if their software will run under VMware\u2019s products or Microsoft\u2019s Desktop Virtualization products, or even Microsoft\u2019s Hyper-V server. Some vendors don\u2019t understand the difference between virtualization and client-server computing. Get any responses in writing.\nRetirement of hardware\nReplacing retired hardware can be a challenge. If it is a stand-alone, isolated product (not connected to anything else), the problem can be resolved by determining the specifications for a replacement, conducting due diligence, etc. It is when data systems, storage, and connections to computers enter the picture that life gets interesting. For example, replacing an instrument sans data system, such as a chromatograph or spectrometer with analog and digital I\/O (sense switches not data) connections to a computer, is essentially just a hardware replacement.\nHardware interfaced to a computer has issues because of software controls and data exchanges. What appears to be the simplest and most common situation is with serial communications (RS-232, RS-xxx). Complications include:\n\n Wiring: Serial communications products do not always obey conventions for wiring, so wiring changes have to be considered and tested.\n Control functions and data exchange: Interacting with serial devices via a computer requires both control functions and data exchange. There are no standards for these, so a new purchase will likely require software changes to these. That may be avoided if the replacement device (e.g., a balance) is from the same vendor as the one you currently have, and is part of a family of products. The vendor may preserve the older command set and add new commands to access new features. If that is the case, you still have a plug-compatible replacement that needs to be tested and qualified for use.\n Interfaces: Moving from an RS-232 or similar RS- device to another interface such as USB will require a new interface (although USB ports are on almost every computer) and software changes.\nIf you are using a USB device, the wiring problems go away but the command structure and data transfer issues remain. Potential software problems are best addressed when the software is first planned and designed; good design means planning ahead for change. The primary issue is control of the external device, as data formats may also change. Those points can be addressed by device-independent programming. That means placing all device-dependent commands in one place\u2014a subroutine\u2014and formatting data into a device-independent format. Doing this makes changes, testing, and other aspects easier.\nLet\u2019s take a single pan balance that has two functions: tare and get_weight. Each of those has a different command sequence of characters that are sent to the balance, and each returns either a completed code or a numerical value that maybe encoded in ASCII, BCD, or binary depending on the vendor\u2019s choice. If the commands to work with the balance are scattered throughout a program, you have a lot of changes to find, make, test, and certify as working. Device-independent programming puts them in two areas: one for the tare command and one for the get_weight command, which returns a floating-point value (e.g., 1.67).\nIf you have to replace the device with a new one, the command codes are changed in two places, and the returned numeric code reformatted into a standard floating point value in one place. The rest of the program works with the value without any concern for its source. That allows for a lot of flexibility in choosing balances in the lab, as different units can be used for different applications with minor software adjustments.\n\nSummary \nAs noted when we first started talking about this goal, the ability to support a product has to be designed and built in, not added on. The issues can be difficult enough when you are working with one vendor. When a second or third vendor is added to the mix, you have an entirely new level of issues to deal with. This is a matter of engineering, not just science. Supportable systems and methods have to be designed, documented, engineered, and validated to be supportable. A system or method isn\u2019t supportable simply because its individual components or steps are.\n\nSeventh goal: Addressing systems integration and harmonization \nIn the 1960s, audio stereo systems came in three forms: a packaged, integrated product that combined an AM\/FM radio tuner, turntable, and speakers; all those components but purchasable individually; and a do-it-yourself, get-out-the-soldering-iron format. The integrated products were attractive because you could just plug them into a power source and they worked. The packaging was attractive, and you didn\u2019t have to know much beyond choosing the functions you wanted to use. In terms of component quality, that was up to the manufacturer; it was rarely top-quality as the components still met the basic needs of the application at a particular price point. \nComponent systems appealed to a different type of customer. They wanted to pick the best components that met their budgets, trading off the characteristics of one item against another, always with the idea of upgrading elements as needed. Each components manufacturer would guarantee that their product worked, but making the entire system work was your issue. In the 1960s, everything was analog so it was a matter of connecting wires. Some went so far as to build the components from kits (lower cost), or design something and get it to work just for the fun of it. HDTV\u2019s have some of the same characteristics as component systems, as they can work out of the box, but if you want better sound or want to add a DVD or streaming box, you have to make sure that the right set of connectors exist on the product and that there is enough of them.\nIn the product cases noted above, there are a limited set of choices for mixing components, so user customization isn't that much of a problem. Most things work together from a hardware standpoint, but software apps are another matter. The laboratory world isn\u2019t quite so neat and tidy.\nFrom a lab standpoint, integrated systems are attractive for a number of reasons:\n\n It suggests that someone has actually thought about how the system should function and what components need to be present, and they put a working package together (may be more than one component).\n It\u2019s installed as a package: when the installation is done, all of it works, both hardware and software.\n It\u2019s been tested as a system and all the components work together.\n You have a single point of contact for support.\n If an upgrade occurs, someone (hopefully) has made sure that upgrading some portions of the system doesn\u2019t mean others are not working; an upgrade embraces the functionality of the entire system.\n The documentation addresses the entire system, from training and support to maintenance, etc.\n It should be easier to work with because the system\u2019s functionality and organization have been thought through.\n It looks nice. Someone designed a packaged system that doesn\u2019t have a number of separate boxes with wires exposed. In the lab, that may be pushing it.\nAchieving that on a component-by-component basis may be a bit of a challenge. Obtaining an integrated system comes down to a few considerations, not the least of which is what you define as a \u201csystem.\u201d A system is a collection of elements that are used to accomplish a task. Figure 21 shows a view of overall laboratory operations.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 21. Laboratory operations and connections to other groups\n\n\n\nLaboratory operations have three levels of systems: the corporate level, the lab administrative level (darker grey, including office work), and the lab bench level. Our concern is going to be primarily with the latter two; however, we have to be aware that the results of lab work (both research and testing) will find their way into the corporate sphere. If the grey box is our system, where does integration fit in and what strategies are available for meeting that goal? Integration is about connecting devices and systems so that there is a smooth flow of command\/control messages, data, and information that does not depend on human intervention. We are not, for example, taking a printout from one instrument system and then manually entering it into another; that transfer should be electronic and bi-directional where appropriate. Why is this important? Improving the efficiency of lab operations, as well as ROI, has long been on the list of desired outcomes from the use of lab automation and computing. Developing integrated systems with carefully designed mechanisms for the flow, storage, and management of K\/I\/D is central to achieving those goals.\nThere are different strategies for building integrated systems like these. One is to create the all-encompassing computer system that does and controls everything. Think HAL in the movie 2001, or popular conceptions of an advanced AI. However, aside from the pitfalls in popular sci-fi, that isn\u2019t an advisable strategy. First, it will most likely never be finished. Trying to come up with a set of functional specifications would take years if they were ever completed. People would be constantly adding features, some conflicting, and that alone (called scope creep) would doom the process, as it has in similar situations. Even if somehow the project were completed, it would reflect the thinking of those involved at the start. In the time that the project was underway, the needs of the lab would change, and the system would be out-of-date as soon as it was turned on. If you were to develop an adaptable system, you'd again still be dealing with scope creep. Other problems would crop up too. If any component needed maintenance, the entire system could be brought to a halt and nothing would get done. Additionally, staff turn-over would be a constant source of delays as new people were brought on board and trained, and as this system would be unique, you couldn't find people with prior experience. Finally, the budget would be hard to deal with, from the initial estimate to the likely overruns.\nAnother approach is to redefine the overall system as a cooperative set of smaller systems, each with its own integration strategy, with the entire unit interconnected. Integrated systems in the lab world are difficult to define as a product set, since the full scope of a lab's processes is highly variable, drawing on a wide range of instruments and equipment. We can define a functional instrument system (e.g., titrators, chromatographic equipment, etc.) but sample prep variability frustrates a complete package. One place that has overcome this is the clinical chemistry market.\nAt this point, we have to pan back a bit and take a few additional aspects of laboratory operations into consideration. Let's look back at Figure 17 again:\n\n\n\n\n\n\n\n\n\n Figure 17. A laboratory process\n\n\n\nWe are reminded that processes are what rule lab work, and the instrumentation and other equipment play an important but subservient role. We need to define the process first and then move on from there; this is the standard validation procedure. In regards to the integration of that instrumentation and equipment, any integration has to support the entire process, not just the individual instrument and equipment packages. This is one of the reasons integration is as difficult as it is. A vendor can create an instrument package, but their ability to put components together is limited to their understanding of their usage.\nUnless the vendor is trying to fit in a well-defined process with enough of a market to justify the work, there is a limit to what they can do. This is why pre-validated products are available in the clinical chemistry market: the process is pre-defined and everyone does the same thing. Note that there is nothing to prevent the same thing happening in other industries; if there were a market for a product that fully implemented a particular ASTM or USP method, vendors might take notice. There is one issue with that, however. When a lab purchases laboratory instrumentation, they often buy general-purpose components. You may purchase an infrared spectrophotometer that covers a wide spectral range that can be used in a variety of applications to justify its cost. Yet you rarely, if ever, purchase one instrument for each lab process that it might be used in unless there was sufficient demand to justify it. And that's the run: if a vendor were to create an equipment package for a specific procedure, the measuring instrument would be stripped down and tailored to the application, and it may not be usable in another process. Is there enough demand for testing to warrant the development of a packaged system? If you were doing blood work, yes, because all blood testing is done the same way; it\u2019s just a question of whether or not your lab is getting enough samples. If it\u2019s ASTM xxx, maybe not.\nAdditionally, the development of integrated systems needs to take the real world into account. Ask whether or not the same equipment can be used for different processes. If the same equipment setup is being used in different processes with different reagents or instrument setups (e.g., different columns in chromatography), developing an integrated electro-mechanical computer system may be a viable project since all the control and electro-mechanical systems would be the same. Returning to the cookie analogy, it's the same equipment, different settings, and different dough mixes (e.g., chocolate chip, sugar cookies, etc.). You just have to demonstrate that the proper settings are being used to ensure that you are getting consistent results.\nIf this sounds a little confusing, it\u2019s because \u201cintegration\u201d can occur on two levels: the movement of data and information and the movement of material. On one hand, we may be talking about integration in the context of merging the sources of data and information (where they are generated) into a common area where they can be used, managed, and accessed according to the lab's and corporate's needs. Along the way, as we\u2019ve seen in the K\/I\/D discussions, different types of data and information will be produced, all of which has to be organized and coordinated. This flow is bi-directional: generated data and information in one direction, and work lists in the other. On the other hand\u2014regarding the movement of materials\u2014we may be talking about automated devices and robotics. Those two hands are joined at the measuring instrument.\nWe\u2019ll begin by discussing the movement of data and information: integration is a matter of bringing laboratory-generated data and information into a structured system where it can be accessed, used, and managed. That \u201csystem\u201d may be a single database or a collection of interconnected data structures. The intent in modern lab integration is that connections between the elements of that structure are electronic, not manual, and transfers may be initiated by user commands or automated processes. Yet note that someone may consider a completely manual implementation process to be \u201cintegrated,\u201d and be willing to accept slower and less efficient data transfers (that\u2019s what we had in the last century). However, that methodology doesn\u2019t give us the improvements in productivity and ROI that are desired.\nThe idea that integration is moving all laboratory K\/I\/D into a structured system is considerably different than the way we viewed things in the past. Previously, the goal was to accumulate lab results into a LIMS or an ELN, with the intermediate K\/I\/D (e.g., instrument data files, etc.) placed in an SDMS. That was a short-sighted attempt at considering only data storage, without fully considering the topic of data utilization. The end goal of using a LIMS or ELN to accumulate lab results was still valid\u2014particularly for further research, reporting, planning, and administrative work\u2014but it didn\u2019t deal effectively with the material those results were based on or the potential need to revisit that work.\nIn this discussion we\u2019ll consider the functional hub of the lab to be a LIMS and\/or ELN, with an optional SDMS; it\u2019s your lab, and you get to choose. We\u2019re referring to this as a hub for a couple of reasons. First, it is the center of K\/I\/D management, as well as planning and administrative efforts. Second, these should be the most stable information systems in the lab; durable and slow to change. Instruments and their data systems will change and be replaced as the lab\u2019s operational needs progress. As a result, the hub is where planning efforts have to begin since decisions made here have a major impact on a lab's ability to meet its goals. The choice of a cloud-based system vs. an on-site system is just one factor to consider.\nHistorically, laboratories have begun by putting the data- and information-generating capability in place first. That\u2019s not unusual, since new companies need data and information to drive their business development. However, what they really need is a mechanism for managing the data and information. Today, however, the place that the development of a laboratory electronic infrastructure needs to begin is with the systems that are used to collect and manage data and information. Then we can put in place the data and information generators. It\u2019s a bit like starting a production line without considering what you\u2019re going to do with all the material you\u2019re producing.\nThe types of data and information generators can vary greatly. Examples include:\n\n a human-based reading that is recorded manually;\n a reading recorded by an instrument with limited storage and communication abilities, e.g., balances and pH meters;\n a reading recorded by a limited-functionality device, where data is recorded and stored but must be transmitted out of the machine to be analyzed; and\n a reading recorded by a combination instrument-computer, which has the ability to record, store, analyze, and output data in various forms.\nThe issue we need to deal with for each of those generators is how to plan for where the output should be stored so that it is accessible and useful. That was the problem with earlier thinking. We focused too much on where the K\/I\/D could be stored and maintained over the long term, but not enough on its ability to be used and managed. Once the analysis was done, we recognized the need to have access to the backup data to support the results, but not the ability to work with it. The next section will look at some of the ramifications of planning for those data types.\nPlanning the integration of your data generators\nThere are several ramifications of planning for your data generators that need to be discussed. Before we begin, though, we need to add two additional criteria for the planning stage:\n\n You should avoid duplication of K\/I\/D unless there is a clear need for it (e.g., a backup).\n In the progression from sample preparation to sample processing, to measurement, to analysis, and then to reporting, there should not be any question of both the provenance and the location of the K\/I\/D generated from that progression.\nThat said, let's look at each generator in greater detail to better understand how we plan for their integration and the harmonization of their resulting K\/I\/D.\n\r\n\n1. A human-based reading that is recorded manually, or recorded from an instrument with limited storage and communication abilities\nExamples: The types of devices we are looking at for these generators are balances, pH meters, volt meters, single-reading spectrophotometers, etc.\nMethod of connection: Both the manual and digital generators don\u2019t leave much choice: the integration is direct data entry into a hub system unless they are being used as part of a LES. Manual modes mean typing (with data entry verification), while digital systems provide for an electronic transfer as the method of integration.\nIssues: There are problems with these generators being directly tied to a hub component (see \u201cA,\u201d Figure 22). Each device or version has its own communications protocol, and the programming is specific to that model. If the item is replaced, the data transfer protocols may differ and the programming has to change. These devices are often used for single readings or weighing a sample, with the result stored in the hub, to be use in later calculations or reporting. However, even though these are single-reading instruments, things can get complicated. Problems may crop up if their measurements are part of a time series, require a weight measurement at specific time intervals, are used for measuring the weights of similar items in medical tablet uniformity testing, or used for measuring pH during a titration. Those applications wouldn\u2019t work well with a direct connection to a hub and would be better served through a separate processor (see \u201cB,\u201d Figure 22) that builds a file of measurements that could be processed, with the results sent to the hub. This creates a need to manage the file and its link to the transmitted results. An SDMS would work well, but the hub system just became more complex. Instead of a device being directly connected to a hub, we have an intermediate system connected to an SDMS and the HUB. Integration is still easily feasible, but more planning is required. Should the intermediate system take on the role of an SDMS (all the files are stored in its file structure), you would also have to provide backup and security facilities to ensure that the files weren\u2019t tampered with and secured against loss. The SDMS would be responsible for entering the results of the work into the hub. (Remember that access to the files is needed to support any questions about the results; printed versions would require re-entering the data to show that the calculations were done properly, which is time-consuming and requires verification.)\nDiagram:\n\n\n\n\n\n\n\n\n\n Figure 22. Representation of the movement of data and information in the lab from a human-based reading that is recorded manually, or from an instrument with limited storage and communication abilities\n\n\n\n\r\n\n2. A reading recorded by a limited-functionality device\nExamples: Measurements are made on one or more samples, with results stored locally, which have to be transmitted to another computer for processing or viewing. Further use may be inhibited until that is done (e.g., the device may have to transmit one set of measurements before the next set can begin). Such devices include microplate readers and spectrophotometers. Some devices can be operated manually from front panel controls, or via network connections through a higher-level controller.\nMethod of connection: Some devices may retain the old RS-232\/422 scheme for serial transmission of measurements and receiving commands, though most have transitioned to USB, Ethernet, or possibly wireless networking.\nIssues: Most of these devices do not produce final calculated results; that work is left to an intermediate process placed between the device and the hub system (Figure 23). As a result, integration depends on those intermediate processes controlling one or more devices, sometimes coordinating with other equipment, calculating final results, and communicating them to the hub system. Measurement files need to be kept in electronic form to make them easier to back up, copy, transmit, and generally work with. If only printed output is available, it should be scanned and an equivalent machine-readable version created and verified. Each experimental run, which may include one or more samples, should have the associated files bundled together into an archive so that all data is maintained in one place. That may be the intermediate processor\u2019s storage or an SDMS with the appropriate organization and indexing capabilities, including links back from the hub. The electronic files may be used to answer questions about how results were produced, re-run an analysis using the original or new set of algorithms, or have the results analyzed as part of a larger study.\nDiagram:\n\n\n\n\n\n\n\n\n\n Figure 23. Representation of the movement of data and information in the lab from a limited-functionality device\n\n\n\n3. A reading recorded by a combination instrument-computer\nExamples: These generators include one-to-one instrument-to-instrument data systems (IDS), many-to-one instrument-to-computer systems, NMR spectroscopy, chromatography, mass spectrometry, thermal analysis, spectrophotometers, etc.\nMethod of connection: There are several means of connection: 1) detector to computer (A\/D), 2) computer to instrument control and accessory devices such as autosamplers (via, e.g., digital I\/O, USB), and 3) computer to centralized hub systems (via, e.g., USB, Ethernet, wireless networks). Integration between the IDS is accomplished through vendor-supported application programming interfaces (APIs) on both sides of the connection.\nIssues: The primary with these generators is managing the data structures that consist of captured detector output files, partially processed data (e.g., descriptors such as peak size, width, area, etc.) computed results, sample information, worklists, and processing algorithms. Some of this material will get transmitted to the hub, but the hub isn't generally designed to incorporate all of it. A portion of it\u2014 the content of a printed report, for example\u2014could be sent to an SDMS. However, the bulk of it has to stay with the IDS since the software needs to interpret and present the sample data contents; the data files by themselves are useless without software to unpack and make sense of them.\nFrom a planning standpoint, you want to reduce the number of IDSs as much as possible. While chromatography is presumably the only technique that offers a choice between one-to-one and many-to-one instruments to computers, hopefully over time that list will expand to provide better data management. Consider three chromatographs, each with its own IDS. If you are looking for data, you have three systems to check, and hopefully each has its own series of unique sample IDs. Three instruments on one IDS is a lot easier to manage and search. You also have to consider backups, upgrades, general maintenance, and cost.\nMoving the instrument data files to an SDMS may not be effective unless the vendor has made provision for it. The problem is data integrity. If you have the ability to move data out of the system and then re-import it, you open up the possibility of importing data that has been edited. Some vendors prohibit this sort of activity.\nDiagram:\n\n\n\n\n\n\n\n\n\n Figure 24. Representation of the movement of data and information in the lab from a combination instrument-computer\n\n\n\nThe above looked at generator types in isolation; however, in reality, devices and instruments are used in combinations, each producing results that have to be maintained and organized. We must look at the data sets that are generated in the course of executing an experiment or method.\nThe procedures associated with an experiment or method can be executed three ways: manually, using a LES, or using a robotics implementation. The real world isn\u2019t so neatly separated; manual and LES implementations may have some steps that use automated tools. The issue we need to address in planning is the creation of an \u201cexperiment data set\u201d that brings all the results produced into one package. Should questions arise about an experiment, you have a data set that can be used as a reference. That \u201cpackage\u201d may be pages in a notebook, a word processor file, or some other log. It should contain all data recorded during a procedure, or, in the case of IDS capture data, file references or pointers to that instrument data or information. You want to be able to pull up that record and be able to answer any questions that may arise about the work.\nAll of that may seem pretty obvious, but there is one point that needs to be addressed: the database structure, including hub systems, IDS file structures, and SDMS all have to be well defined before you accumulate a number of experiment packages. You don\u2019t want to find yourself in a situation where you have a working system of data or information storage and then have to make significant changes to it. That could mean that all previous packages have to be updated to reflect the new system, or, worse, have to deal with an \u201cold\u201d and \u201cnew\u201d system of managing experimental work.\nLES systems come in two forms: stand-alone software packages, and script-based systems that are part of a LIMS or ELN. The stand-alone systems should produce the experiment record automatically with all data and pointers to IDS captured data or information. For script-based systems, the programming for the LES function has to take that into account. As for laboratory robotics, they can be viewed as an extension of a LES: instead of a person following instructions, a robot or a collection of robotic components follows its programming to carry out a process. Developing an experimental record is part of that process.\nThe bottom line in all of this is simple: the management architecture for your K\/I\/D has to be designed deliberately and put in place early in a lab's development. If it is allowed to be created on an as-needed basis, the resulting collection of computers and storage will be difficult to maintain, manage, and expand in an orderly fashion. At some point, someone is going to have to reorganize it, and that will be an expensive and perhaps painful process.\n\nHarmonization \nHarmonization is a companion goal to integration. Approached with the right mindset it can reduce:\n\n installation costs,\n support costs,\n education and training requirements, and\n development effort.\nHarmonization efforts, if used inappropriately, can create strife, increasing inter-departmental friction and conflict. The general idea of harmonization is to use common hardware and software platforms to implement laboratory systems, while ensuring that move toward commonality doesn\u2019t force people to use products that are force-fits, that don\u2019t really meet the lab's needs, but serve some other agenda. The purpose of computing systems is to help people get their work done; if they need a specific product to do it, end of story. If it can be provided using common hardware and software platforms, great, but that should not be a limiting factor. If used as a guide in the development of database systems, harmonization can make it easier to access laboratory K\/I\/D across labs. It may slow down implementations because more people\u2019s opinions have to be taken into account, but the end result will be the ability to gain more use out of your K\/I\/D.\nFigure 25 shows a common LIMS server supporting three different labs. Each lab has its own database structure, avoiding conflicts and unnecessary compromises in the conduct of lab work. It does benefit reduced implementation costs and support costs. While some vendors support this, others may not; see if they are willing to work a deal since there are multiple labs systems involved. If we couple this with the common structure definitions of K\/I\/D noted earlier, accessing information across labs will be more productive.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 25. Harmonizing LIMS platforms\n\n\n\nAn alternative is to force everyone into one data structure, usually to reduce costs. Savings on licensing costs may be offset by development delays as multiple labs resolve conflicts in database organization, security, access control, etc. In short, keep it simple; things will work smoother and in the long run be less costly from an implementation, maintenance, and support perspective. If there is a need or desire to go through the databases for accounting purposes or other organizational requirements, the necessary material can be exported into another file structure that can be analyzed as needed. This provides a layer of security between the lab and the rest of the organization. It\u2019s basically a matter of planning how database contents are being managed with the lab and what has to be accessed from other parts of the organization.\nPart of harmonization planning process involves examining how computers are paired with instruments. You may not have multiple instances of higher-priced equipment such as mass spectrometers, NMRs, or other instruments, and having a computer dedicated to each device makes sense. However there is one instrument that you may have several of: chromatographs. You can purchase a computer for each instrument, but in this case, most CDS can support multiple instruments (Figure 26).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 26. Consolidate instrument-computer connections where feasible\n\n\n\nThere are advantages to having multiple instruments on one computer:\n\n You have only one system to support, maintain, and back up.\n All the K\/I\/D is in one system.\n The qualification or validation process is performed once, rather than having it repeated for each system.\n Overall cost is reduced.\nThis \"multiple instruments to one computer\" configuration is the result of a low data collection rate, the modest computing requirements needed to process the instrument data, and user demands on the vendors. Given the developments in computing power and distributed data acquisition and control, this many-to-one configuration should be extended to other instrument techniques, reducing costs and bringing more efficiency to the management of K\/I\/D.\nRegarding computer systems...\nHarmonization doesn't mean that everything should run the same OS or the same version of the OS. It means doing it where possible, but not at the expense of doing lab work effectively.\nWith the wide diversity of products in the laboratory market, you\u2019re going to find a mix of large and small vendors. Some may be small, growing companies that are managed by a few people, and as a result, keeping up with the latest versions of operating systems and underlying software may not be critical if it doesn\u2019t affect their product's usability or performance. Their product certification on the latest version of a software platform may lag larger vendors. That means that requiring all systems to be at the same operating system level isn\u2019t realistic. Upgrading the OS may disable the software that lab personnel depend upon.\nRegarding the data...\nDuring November 19-20, 2019 Pharma IQ\u2019s Laboratory Informatics Summit held a meeting on \"Data Standardization for Lab Informatics.\" The meeting highlighted the emerging FAIR Guiding Principles, which state that K\/I\/D should be findable, accessible, interoperable, and reusable (FAIR). The point of mentioning this is to highlight the growing, industry-wide importance of protecting the value of the K\/I\/D that you are collecting. No matter how much it costs to produce, if you can\u2019t find the K\/I\/D you need, it has no value because it isn\u2019t usable. The same holds true if the data supporting information can\u2019t be found.\nUtilization is at the core of much of what we\u2019ve been discussing. Supporting the FAIR Guiding Principles should be part of every discussion about products and what they produce, how the database is designed, and what the interoperability between labs in your organization looks like.\nAnother aspect of this subject is harmonizing data definitions across your organization. The same set of terms should be used to describe an object or aspect, and their database representation should be compatible, etc. The point is to make it easier to find something and make use of it.\nPutting this all to use\nHow do you apply all of this to your new lab (an easier task) or existing lab (more challenging)? This is going to be a broad-brush discussion since every lab has their own way of handling things, from its overall mission to its equipment and procedures, so you\u2019re going to have to take these points and adjust them to fit your requirements.\nTo start, assume you have a hub system (either a LIMS or ELN) as the center of gravity for all your K\/I\/D collection. You build your lab's K\/I\/D management infrastructure from this center of gravity outward; effectively everything revolves around the hub and radiates out from it.[l] \nFor each K\/I\/D generator, ask:\n\n What does it produce, which of the K\/I\/D generator types noted earlier matches, and which model is appropriate? \n Does it generate a file that has to be processed, or is it the final measurement? \n Does the device or the system supporting it have all the information needed to move it on to the next phase of the process? For example if the device is a pH meter, what is going to key the result into the next step? It will need a sample or experiment reference ID so that it knows where the result should go.\nFor each device output, ask:\n\n What happens to the generated K\/I\/D and how is it used? And remember, nothing should ever get deleted.\n Is the device output a single measurement or part of a set? Will it be combined with measurements from other devices, sample IDs, and calibration information?\n Where is the best place to put it? In an intermediate server, SDMS, or hub?\n If it is a final result of that process, should it be in the hub?\n If it is an intermediate file or result, then where?\n How might it be used in the future and what is the best way to prepare for that? Files may need to be examined in audits, transferred to another group or organization[m], or recalculated with new algorithms. Does your system provide trace-back from final results to source data?\nFor all devices, ask: \n\n Does every device that has storage and communications capability have back up procedures put in place?\nDepending on your point of view, whether it is on the science, laboratory operations management, or lab administration, your interest in lab computing may range from \u201cnecessary evil\u201d to \u201cmakes life easier\u201d to \u201cneeded to make the lab function,\u201d or some other perspective. You may be of the opinion that all this is interesting but not your responsibility. If not yours, then who? That topic will be covered in the next write-up.\n\nLaboratory systems engineers \nIf you believe that the technology planning and management considerations noted so far in this guide are important to your laboratory, it's time to ask to whom that responsibility falls upon?\nThe purpose of this guide has been to highlight that the practice of science has changed, become more complex, and become more dependent on technologies that demand a lot of attention. Those technologies are not only the digital systems we\u2019ve covered, but also the scientific methodologies and instrumentation whose effective use can take\u2014through increasing specialization and depth of material\u2014an entire career to learn and apply. The user of scientific computing typically views it as a tool for getting work done and not another career. Once upon a time, the scientist knowledgeable in both laboratory work and computing was necessary; if you wanted to use computers you had to understand how they worked. Today, if you tried to do that, you\u2019d find yourself spread thin across your workload, with developments happening faster in science and computers than you a single individual can keep up with.\nLet's look at what a laboratory scientist would need to be able to do in order to also support their laboratory's scientific computing needs, in addition to their normal tasks (Figure 27).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 27. Partial task list for supporting laboratory scientific computing\n\n\n\nOn top of those tasks, the lone scientist would also have to have the following technological knowledge and personal capabilities (Figure 28):\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 28. Partial list of technological knowledge and personal capability requirements for supporting laboratory scientific computing\n\n\n\nLooking at these two figures, we're realistically considering two levels of expertise: a high, overview level that can look at the broader issues and see how architectures can be constructed and applied, and specialists in areas such as robotic, etc. However, the current state of undergraduate\u2014and to a lesser extent graduate\u2014education doesn\u2019t typically have room for the depth of course work needed to cover the material noted above. Expanding your knowledge base into something that is synergistic with your current course work is straightforward; doing it with something that is from a separate discipline creates difficulties. Where do digital systems fit into your life and laboratory career?\nLet's look at what the average laboratory scientist does today. Figure 29 shows the tasks that are found in modern laboratory operations in both research and testing facilities. \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 29. Typical laboratory activities in any scientific discipline\n\n\n\nNow let's compare that set of tasks in Figure 29 with the task emphasis provided in today's undergraduate laboratory science courses and technology courses (Figure 30 and 31).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 30. Task-level emphasis of laboratory science in higher-education courses\n\n\n\n\n\n\n\n\n\n\n\n\n Figure 31. Task-level emphasis of information technology in higher-education courses\n\n\n\nIn cases where science students have access to instrumentation-computer systems, the computers are treated as \u201cblack boxes\u201d that acquire the data (data capture), process it (data processing) and report it. How those things happen is rarely if ever discussed, with no mention of analog-digital converters, sampling rates, analysis algorithms, etc. \u201cStuff happens,\u201d yet that \u201cstuff,\u201d if not properly ran with tested parameters, can turn good bench science into junk data. How would they know? Students may or may not get exposure to LIMS or ELN systems even though it would be useful for students to capture and work with their lab results, but schools may not be willing to invest in them.\nIT students will be exposed to data and information management through database courses, but not at the level that LIMS and ELNs require (e.g., instrument communications and control); the rest of the tasks in Figure 31 is practically unknown to them. They\u2019d be happy to work on the computer in Figure 31, but the instrument and the instrument connections\u2014the things that justify the computer's role\u2014aren\u2019t something they\u2019d be exposed to.\nWhat we need are people with a foot in both fields, able to understand and be conversant in both the laboratory science and IT worlds, relating them to each other to the benefit of lab operation effectiveness while guiding IT in performing their roles. We need \u201claboratory systems engineers\u201d (LSEs).\nPreviously referred to as \"laboratory automation engineers\" (LAEs)[6] and \"LAB-IT\" specialists, we now realize both titles fall short of the mark. \"Laboratory automation engineer\" emphasizes automation too strongly when the work is much broader than that. And \"LAB-IT\" is a way of nudging IT personnel into lab-related work without really addressing the full scope of systems that exist in labs, including robotics and data acquisition and control.\nLaboratory information technology support differs considerably from classical IT work (Figure 32). The differences are primarily two-fold. First, the technologies used in lab work, including those in which instruments are attached to computers and robotics, are different than those commonly encountered in classical IT. The computers are the same, but the added interface and communications requirements imposed by instrument-computer connections change the nature of the work. When troubleshooting, it can be difficult to separate computer issues from those resulting from the connection to instruments and digital control systems. Second, the typical IT specialist, maybe straight out of school, doesn\u2019t have a frame of reference for understanding what they are dealing with in a laboratory setting. The work is foreign, the discussions involve terminology they may not understand, and there may be no common ground for discussing problems. In classical IT, the IT personnel may be using the same office software as the people they support, but they can't say the same for the laboratory software used by scientists.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 32. Comparison of corporate IT with laboratory IT (LAB-IT)\n\n\n\nHaving noted the differences between classing IT and laboratory IT, as well as the growing need for competent LSEs, we need to take a closer look at some of the roles that classic IT and LSE personnel can take. Figure 33 provides a sub-set of the items from Figure 32 and reflects tasks that IT groups could be comfortable with. Typical IT backgrounds with no lab tech familiarity won\u2019t get you beyond the basic level of support. To be effective, IT personnel need to become familiar with the lab environment, the applications and technologies used, and the language of laboratory work. It isn\u2019t necessary for IT support to become experts in instrumental techniques, but they should understand the basic \"instrument to control system to computer\" model as well as the related database applications, to the point where they can provide support, advise people on product selections, etc. We need people who can straddle the IT-laboratory application environment. They could be lab people with an interest in computing or IT people with a strong interest in science.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 33. Potential roles for IT and LSE support in laboratory work\n\n\n\nThere are ways of bridging that education gap (Figure 34), but today they depend upon individual initiative more than corporate direction to educate people to the level needed. On-the-job training is not an effective substitute for real education; on the surface it is cheaper, but you lose out in the long run because people really don\u2019t understand what is going on, which limits their effectiveness and prevents them from being innovative or even catching problems in the early stages before they become serious. A big issue is this: due to a lack of education, are people developing bad K\/I\/D and aren't aware of it? The problem isn\u2019t limited to the level of systems we are talking about here. It also extends to techniques such as pipetting.[15]\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 34. Bridging the education gap\n\n\n\nIt is also a matter of getting people to understand the breadth of material they have to be familiar with. In 2018, a webinar series was created (Figure 35) to educate management on the planning requirements for implementing lab systems. The live sessions were well attended. The chart shows the viewing rate for the individual topics through early December 2020. Note that the highest viewed items were technology-specific; people wanted to know about LIMS, ELN, etc. The details about planning, education, support, etc. haven\u2019t received near the amount of attention they need. People want to know about product classes but aren\u2019t willing to learn about what it takes to be successful. Even if you are relying on vendors or consultants, lab management is still accountable for the success of planning, implementation, and effectiveness of lab systems.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 35. Laboratory technology webinar series views after initial release\n\n\n\nPrior to the COVID-19 pandemic of 2020, undergraduate education depended on the standard model of in-person instruction. With the challenges of COVID-19 spreading, online learning took on a new importance and stronger acceptance, building on the ground established by online universities and university programs. This gives us an acceptable model for two types of course development: a fully-dedicated LSE program or an expanded program that would expand student\u2019s backgrounds in both the laboratory sciences and IT. One issue that would need to be addressed, however, is bridging the gap between presentation material and hands-on experience with lab systems. Videos and evaluation tests will only get you so far; you need the hands-on experience to make it real and provide the confidence that what you've learned can be effectively applied.\nThere are several steps that can be taken to build an LSE program. The first is to develop a definition of a common set of skills and knowledge that an LSE should have, recognizing that people will come from two different backgrounds (i.e., laboratory science and IT), and those have to be built up to reach a common balanced knowledge base. Those with a strong laboratory science background need to add information technology experience, while those from IT will need to gain an understanding of how laboratory science is done. Remember, however, that those from IT experiences don\u2019t need to be educated in chemistry, biology, and physics, etc. After all, they aren\u2019t going to be developing methods; they will be helping to implement them. There are things common to all sciences that they need to understand such as record keeping, the workflow models of testing and research, data acquisition and processes, instrumentation, and so on. That curriculum should also help people who want to specialize in particular subject areas such as laboratory database systems, robotics, etc. The second step is to build a curriculum that allows students to meet those requirements. This requires solid forethought in the development and curation of course materials. A lot of material already exists and is spread over the internet on university, government, and company web sites. A good first step would be to collect and organize those references into a single site (the actual courses need not be moved to the site, just their descriptions, access requirements, and links). Presentation and organization of the content is also important. Someone visiting the site will need a guide of what LSE is about, how to find material appropriate for different subject areas, and how to get access to it. Consider your site audience a visitor that knows nothing about the field: where do they start and how do we facilitate their progress? Providing clear orientation and direction are key. First give them an understanding of what LSE is all about, and then a map to whatever interests them. With the curriculum built, you can then identify areas that need more material and then move to further develop the program. Of course, you'll also want to make it possible to take advantage of online demonstration systems and simulators to give people a feel for working with the various laboratory systems. This is a half-step to what is needed: there\u2019s no substitute for hands-on work with equipment.\nAs it stands today, we\u2019ve seemingly progressed from manual methods, to computer-assisted methods, and then to automated systems in the course of developing laboratory technologies over the years, and yet our educational programs are a patchwork of courses largely driven by individual needs. We need to take a new look at lab technologies and their use and how best to prepare people for their work with solid educational opportunities.\n\nClosing \nThis guide has addressed the following:\n\n Why technology planning and management needs to be addressed: Because integrated system need attention in their application and management to protect electronic laboratory K\/I\/D, ensure that it can be effectively used, and ensure that the systems and products put in place are both the right ones, and that they fully contribute to improvements in lab operations.\n What's changed about that planning and management since the introduction of computers in the lab: As technology in the lab expanded, we lost the basic understanding of what the new computer and instrument system were and what they did, that they had faults, and that if we didn\u2019t plan for their effective use and counter those faults, we were opening ourselves to unpleasant surprises. The consequences at times were system crashes, lost data, and a lack of a real understanding of how the output of an instrument was transformed into a set of numbers, which meant we couldn\u2019t completely account for the results we were reporting. A more purposeful set of planning and management activities, at the earliest point possible, have become increasingly more important.\n Why developing an environment that fosters productivity and innovation is important: Innovation doesn\u2019t happen in a highly structured environment: you need the freedom to question, challenge, etc. You also need the tools to work with. The inspiration that leads to innovation can happen anywhere, anytime. All of a sudden all the pieces fit. This requires flexibility and trust in people, an important part of corporate culture.\n Why developing high-quality K\/I\/D is desirable: There are different types of data structures that are used in lab work, and careful attention is needed to work with and manage them. This includes the effective management of K\/I\/D, putting it in a structure that encourages its use and protects its value. When methods are proven and you have documented evidence that they were executed by properly educated personnel using qualified reagents, instruments, and methods, you should then have high-quality K\/I\/D to support each sample result and any other information gleaned from that data.\n Why fostering a culture around data integrity is important to lab operations, addressing both technical and personnel issues: Positive outcomes will come from your data integrity efforts: your work will be easier and protected from loss, results will be easier to organize and analyze, and you\u2019ll have a better functioning lab. You\u2019ll also have fewer unpleasant surprises when technology changes occur and you need to transition from one way of doing things to another.\n How to address digital, facility, and backup security: Preventing unauthorized electronic and physical intrusion is critical to data integrity and meeting regulatory requirements. It also ensures that access to K\/I\/D is protected against loss from a wide variety of threats to the organization's facilities, all while securing your ability to work. This included addressing power backup, continuity of operations, systems backup, and more.\n How to acquire and develop \"products\" that support regulatory requirements: Careful engineering and well-planned and -documented internal processes are needed to ensure that systems and methods that are being used can remain in use and be supported over the life span of a lab. This means recognizing the initial design and planning of processes and methods has to be done well for a supportable product, and keeping in mind the potential for future process review and modification even as the initial process or method is being developed. Additionally, the lab must also recognize the complete product life cycle and how that affects the supportability of systems and methods.\n The importance of system integrations and the harmonization of K\/I\/D: Integrated systems can benefit a lab's operations and the planning needed to work with different types of database systems as the results of lab work becoming more concentrated in LIMS and ELNs, including making decisions about how K\/I\/D is stored and distributed over multiple databases. At the same time, harmonization efforts using common hardware and software platforms to implement laboratory systems is important, but those efforts must also ensure that the move toward commonality doesn\u2019t force people to use products that are forced fits, that don\u2019t really meet the lab's needs, but serve some other agenda. \n Why the development of comprehensive higher-education courses dedicated to the laboratory systems engineer or lab science-IT hybrid is a must with today's modern laboratory technology: In today's world, typical IT backgrounds with no lab tech familiarity, or typical laboratory science backgrounds with no IT familiarity won\u2019t get you beyond the basic level of support for your laboratory systems. To be effective, IT personnel need to become familiar with the lab environment, the applications and technologies used, and the language of laboratory work, while scientists must become more familiar with the management of K\/I\/D from the technical perspective. This gap must be closed through new and improved higher-education programs.\nAnd thus we return back to the start to close this guide. First, there's a definite need for better planning and management of laboratory technologies. Careful attention is required in to protect electronic laboratory knowledge, information, and data (K\/I\/D), ensure that it can be effectively used, and ensure that the systems and products put in place are both the right ones, and that they fully contribute to improvements in lab operations. Second, seven clear goals highlight this apparent need for laboratory technology planning and management and improve how it's performed. From supporting an environment that fosters productivity and innovation all the way to ensuring proper systems integration and harmonization, planning and management is a multi-step process with many clear benefits. And finally, there's a definitive need for more laboratory systems engineers (LSEs) who have the education and skills needed to accomplish all that planning and management in an effective manner, from the very start. This will require a more concerted effort in academia, and perhaps even among professional organizations catering to laboratories. All of this together hopefully means a more thoughtful, modern, and deliberate approach to implementing laboratory technologies in your lab.\n\nAbbreviations, acronyms, and initialisms \nA\/D: Analog-to-digital\nAI: Artificial intelligence\nALCOA: Attributable, legible, contemporaneous, original, and accurate\nAPI: Application programming interface\nCDS: Chromatography data system\nCPU: Central processing unit\nELN: Electronic laboratory notebook\nEPA: Environmental Protection Agency\nFAIR: Findable, accessible, interoperable, and reusable\nFDA: Food and Drug Administration\nFRB: Fast radio bursts\nIT: Information technology\nISO: International Organization for Standardization\nK\/D\/I: Knowledge, data, and information\nLAB-IT: Laboratory information technology support staff\nLAE: Laboratory automation engineering (or engineer)\nLES: Laboratory execution system\nLIMS: Laboratory information management system\nLIS: Laboratory information system\nLOF: Laboratory of the future\nLSE: Laboratory systems engineer\nML: Machine learning\nOS: Operating system\nQA\/QC: Quality assurance\/quality control\nROI: Return on investment\nSDMS: Scientific data management system\nSOP: Standard operating procedure\nTPM: Technology planning and management\n\r\n\n\nFootnotes \n\n\n\u2191 See Elements of Laboratory Technology Management and the LSE material in this document. \n\n\u2191 See the \"Scientific Manufacturing\" section of Elements of Laboratory Technology Management. \n\n\u2191 By \u201cgeneral systems\u201d I\u2019m not referring to simply computer systems, but the models and systems found under \u201cgeneral systems theory\u201d in mathematics. \n\n\u2191 Regarding LAB-IT and LAEs, my thinking about these titles has changed over time; the last section of this document \u201cLaboratory systems engineers\u201d goes into more detail. \n\n\u2191 \u201cReal-time\u201d has a different meaning inside the laboratory than it does in office applications. Instead of a response of a couple seconds between an action and response, lab \u201creal-time\u201d is often a millisecond or faster precision; missing a single sampling timing out of thousands can invalidate an entire sample analysis. \n\n\u2191 See Notes on Instrument Data Systems for more on this topic. \n\n\u2191 For a more detailed description of the K\/I\/D model, please refer to Computerized Systems in the Modern Laboratory: A Practical Guide. \n\n\u2191 For more detailed discussion on this, see Notes on Instrument Data Systems. \n\n\u2191 For more information on virtualization, particularly if the subject is new to you, look at Next-Gen Virtualization for Dummies. The for Dummies series is designed to educate people new to a topic, getting away from jargon and presenting material in clear, easy-to-understand language. This book is particularly good at that. \n\n\u2191 One good reference on this subject is a presentation Building A Data Integrity Strategy To Accompany Your Digital Enablement by Julie Spirk Russom of BioTherapeutics Pharmaceutical Science. \n\n\u2191 Though it may not see significant updates, consider reading the Comprehensive Guide to Developing and Implementing a Cybersecurity Plan for a much more comprehensive look at security in the lab. \n\n\u2191 Yes the scientific work you do is essential to the lab\u2019s purpose, but our focus is on one element of the lab\u2019s operations: what happens after the scientific work is done. \n\n\u2191 For example, a product line is sold to another company or transferred to another division, and they then want copies of all relevant information. Meeting regulatory requirements is another example. \n\n\nAbout the author \nInitially educated as a chemist, author Joe Liscouski (joe dot liscouski at gmail dot com) is an experienced laboratory automation\/computing professional with over forty years of experience in the field, including the design and development of automation systems (both custom and commercial systems), LIMS, robotics and data interchange standards. He also consults on the use of computing in laboratory work. He has held symposia on validation and presented technical material and short courses on laboratory automation and computing in the U.S., Europe, and Japan. He has worked\/consulted in pharmaceutical, biotech, polymer, medical, and government laboratories. His current work centers on working with companies to establish planning programs for lab systems, developing effective support groups, and helping people with the application of automation and information technologies in research and quality control environments.\n\nReferences \n\n\n\u2191 Bourne, D. (2013). \"My boss the robot\". Scientific American 308 (5): 38\u201341. doi:10.1038\/scientificamerican0513-38. PMID 23627215.   \n\n\u2191 Cook, B. (2020). \"Collaborative Robots: Mobile and Adaptable Labmates\". Lab Manager 15 (11): 10\u201313. https:\/\/www.labmanager.com\/laboratory-technology\/collaborative-robots-mobile-and-adaptable-labmates-24474 .   \n\n\u2191 Hsu, J. (24 September 2018). \"Is it aliens? Scientists detect more mysterious radio signals from distant galaxy\". NBC News MACH. https:\/\/www.nbcnews.com\/mach\/science\/it-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586 . Retrieved 04 February 2021 .   \n\n\u2191 Timmer, J. (18 July 2018). \"AI plus a chemistry robot finds all the reactions that will work\". Ars Technica. https:\/\/arstechnica.com\/science\/2018\/07\/ai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work\/5\/ . Retrieved 04 February 2021 .   \n\n\u2191 \"HelixAI - Voice Powered Digital Laboratory Assistants for Scientific Laboratories\". HelixAI. http:\/\/www.askhelix.io\/ . Retrieved 04 February 2021 .   \n\n\u2191 6.0 6.1 Liscouski, J.G. (2006). \"Are You a Laboratory Automation Engineer?\". SLAS Technology 11 (3): 157-162. doi:10.1016\/j.jala.2006.04.002.   \n\n\u2191 Liscouski, J. (2015). \"Which Laboratory Software Is the Right One for Your Lab?\". PDA Letter (November\/December 2015): 38\u201341. https:\/\/www.researchgate.net\/publication\/291971749_Which_Laboratory_Software_is_the_Right_One_for_Your_Lab .   \n\n\u2191 \"Data integrity\". Wikipedia. 03 February 2021. https:\/\/en.wikipedia.org\/wiki\/Data_integrity . Retrieved 07 February 2021 .   \n\n\u2191 Harmon, C. (20 November 2020). \"What Is Data Integrity?\". Technology Networks. https:\/\/www.technologynetworks.com\/informatics\/articles\/what-is-data-integrity-343068 . Retrieved 07 February 2021 .   \n\n\u2191 Chrobak, U. (17 August 2020). \"The US has more power outages than any other developed country. Here\u2019s why\". Popular Science. https:\/\/www.popsci.com\/story\/environment\/why-us-lose-power-storms\/ . Retrieved 08 February 2021 .   \n\n\u2191 Tulsi, B.B. (04 September 2019). \"Greater Awareness and Vigilance in Laboratory Data Security\". Lab Manager. https:\/\/www.labmanager.com\/business-management\/greater-awareness-and-vigilance-in-laboratory-data-security-776 . Retrieved 09 February 2021 .   \n\n\u2191 Riley, D. (21 May 2020). \"GitLab runs phishing test against employees \u2013 and 20% handed over credentials\". Silicon Angle. https:\/\/siliconangle.com\/2020\/05\/21\/gitlab-runs-phishing-test-employees-20-handing-credentials\/ . Retrieved 09 February 2021 .   \n\n\u2191 \"Automated Sample Preparation\". Fluid Management Systems, Inc. https:\/\/www.fms-inc.com\/sample-prep\/ . Retrieved 09 February 2021 .   \n\n\u2191 \"PAL Auto Sampler Systems\". Agilent Technologies, Inc. https:\/\/www.agilent.com\/en\/product\/gas-chromatography\/gc-sample-preparation-introduction\/pal-auto-sampler-systems . Retrieved 09 February 2021 .   \n\n\u2191 Bradshaw, J.T. (30 May 2012). \"The Importance of Liquid Handling Details and Their Impact on Your Assays\" (PDF). European Lab Automation Conference 2012. Artel, Inc. https:\/\/d1wfu1xu79s6d2.cloudfront.net\/wp-content\/uploads\/2013\/10\/The-Importance-of-Liquid-Handling-Details-and-Their-Impact-on-Your-Assays.pdf . Retrieved 11 February 2021 .   \n\n\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\">https:\/\/www.limswiki.org\/index.php\/LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering<\/a>\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tLII\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 18 February 2021, at 19:33.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,756 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","655f7d48a642e9b45533745af73f0d59_html":"<body class=\"mediawiki ltr sitedir-ltr ns-202 ns-subject page-LII_Laboratory_Technology_Planning_and_Management_The_Practice_of_Laboratory_Systems_Engineering skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">LII:Laboratory Technology Planning and Management: The Practice of Laboratory Systems Engineering<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\"><p><b>Title<\/b>: <i>Laboratory Technology Planning and Management: The Practice of Laboratory Systems Engineering<\/i>\n<\/p><p><b>Author for citation<\/b>: Joe Liscouski, with editorial modifications by Shawn Douglas\n<\/p><p><b>License for content<\/b>: <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/creativecommons.org\/licenses\/by\/4.0\/\" target=\"_blank\">Creative Commons Attribution 4.0 International<\/a>\n<\/p><p><b>Publication date<\/b>: December 2020\n<\/p>\n\n\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>What separates successful advanced <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratories<\/a> from all the others? It's largely their ability to meet their goals, with the effective use of resources: people, time, money, equipment, data, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>. The fundamental goals of laboratory work haven\u2019t changed, but they are under increased pressure to do more and do it faster, with a better return on investment (ROI). Laboratory managers have turned to electronic technologies (e.g., computers, networks, robotics, microprocessors, database systems, etc.) to meet those demands. However, without effective planning, technology management, and education, those technologies will only get labs part of the way to meeting their needs. We need to learn how to close the gap between getting part-way there and getting where we need to be. The practice of science has changed; we need to meet that change to be successful.\n<\/p><p>This document was written to get people thinking more seriously about the technologies used in laboratory work and how those technologies contribute to meeting the challenges labs are facing. There are three primary concerns:\n<\/p>\n<ol><li> The need for planning and management: When digital components began to be added to lab systems, it was a slow incremental process: integrators and microprocessors grew in capability as the marketplace accepted them. That development gave us the equipment we have now, equipment that can be used in isolation or in a networked, integrated system. In either case, they need attention in their application and management to protect electronic laboratory data, ensure that it can be effectively used, and ensure that the systems and products put in place are both the right ones, and that they fully contribute to improvements in lab operations.<\/li>\n<li> The need for more laboratory systems engineers (LSEs): There is increasing demand for people who have the education and skills needed to accomplish the points above and provide research and testing groups with the support they need.<sup id=\"rdp-ebb-cite_ref-1\" class=\"reference\"><a href=\"#cite_note-1\">[a]<\/a><\/sup><\/li>\n<li> The need to collaborate with vendors: In order to develop the best products needed for laboratory work, vendors should be provided more user input. Too often vendors have an idea for a product or modifications to existing products, yet they lack a fully qualified audience to bounce ideas off of. With the planning in the first concern in place, we should be able to approach vendors and say, with confidence, \"this is what is needed\" and explain why.<\/li><\/ol>\n<p>If the audience for this work were product manufacturing or production facilities, everything that was being said would have been history. The efficiency and productivity of production operations directly impacts profitability and customer satisfaction; the effort to optimize operations would have been an essential goal. When it comes to laboratory operations, that same level of attention found in production operations must be in place to accelerate laboratory research and testing operations, reducing cost and improving productivity. Aside from a few lab installations in large organizations, this same level of attention isn\u2019t given, as people aren\u2019t educated as to its importance. The purpose of this work is to present ideas of what laboratory technology challenges can be addressed through planning activities using a series of goals.\n<\/p><p>This material is an expansion upon two presentations:\n<\/p>\n<ul><li> \"<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13140\/RG.2.2.24722.40645\" target=\"_blank\">Laboratory Technology Management & Planning<\/a>,\" 2nd Annual Lab Asset & Facility Management in Pharma 2019, San Diego, CA, October 22, 2019<\/li>\n<li> \"<a rel=\"external_link\" class=\"external text\" href=\"https:\/\/go.labmanager.com\/webinar-2020-digital-technologies\" target=\"_blank\">How Digital Technologies are Changing the Landscape of Lab Operations<\/a>,\" <i>Lab Manager<\/i> webinar, April 2020<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Directions_in_lab_operations\">Directions in lab operations<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"The_lab_of_the_future\">The lab of the future<\/span><\/h4>\n<p>People often ask what the lab of the future (LOF) is going to look like, as if there were a design or model that we should be aspiring toward. There isn\u2019t. Your lab's future is in your hands to mold, a blank sheet of paper upon which you define your lab's future by setting objectives, developing a functional physical and digital architecture, planning processes and implementations, and managing technology that supports both scientific and laboratory <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">information management<\/a>. If that sound scary, it\u2019s understandable. But you must take the time to educate yourself and bring in people (e.g., LSEs, consultants, etc.) who can assist you.\n<\/p><p>Too often, if vendors and consultants are asked what the LOF is going to look like, the response lines up with their corporate interests. No one knows what the LOF is because there isn\u2019t a singular future, but rather different futures for different types of labs. (Just think of all the different scientific disciplines that exist; one future doesn\u2019t fit all.) Your lab's future is in your hands. What do you want it to be?\n<\/p><p>The material in this document isn\u2019t intended to define your LOF, but to help you realize it once the framework has been created, and you are in the best position to create it. As you create that framework, you'll be asking:\n<\/p>\n<ol><li> Are you satisfied with your lab's operations? What works and what doesn\u2019t? What needs fixing and how shall it be prioritized?<\/li>\n<li> Has management raised any concerns?<\/li>\n<li> What do those working in the lab have to say?<\/li>\n<li> How is your lab going to change in the next one to five years?<\/li>\n<li> Does your industry have a working group for lab operations, computing, and automation?<\/li><\/ol>\n<p>Adding to question five, many companies tend to keep the competition at arm's length, minimizing contact for fear of divulging confidential information. However, if practically everyone is using the same set of test procedures from a trusted neutral source (e.g., <a href=\"https:\/\/www.limswiki.org\/index.php\/ASTM_International\" title=\"ASTM International\" class=\"wiki-link\" data-key=\"dfeafbac63fa786e77b472c3f86d07ed\">ASTM International<\/a>, United States Pharmacopeia, etc.), there\u2019s nothing confidential there. Instead of developing automated versions of the same procedure independently, companies can join forces, spread the cost, and perhaps come up with a better solution. With that effort as a given, you collectively have something to approach the vendor community with and say \u201cwe need this modification or new product.\u201d This is particularly beneficial to the vendor when they receive a vetted product requirements document to work from.\n<\/p><p>Again, you don\u2019t wait for the lab of the future to happen, you create it. If you want to see the direction lab operations in the future can take, look to the manufacturing industry: it has everything from flexible manufacturing, cooperative robotics<sup id=\"rdp-ebb-cite_ref-BourneMyBoss13_2-0\" class=\"reference\"><a href=\"#cite_note-BourneMyBoss13-2\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CookCollab20_3-0\" class=\"reference\"><a href=\"#cite_note-CookCollab20-3\">[2]<\/a><\/sup>, and so on.<sup id=\"rdp-ebb-cite_ref-4\" class=\"reference\"><a href=\"#cite_note-4\">[b]<\/a><\/sup> This is appropriate in both basic and applied research, as well as <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a>.\n<\/p><p>Both manufacturing and lab work are process-driven with a common goal: a high-quality product whose quality can be defended through appeal to process and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">data integrity<\/a>.\n<\/p><p>Lab work can be broadly divided into two activities, with parallels to manufacturing: experimental procedure development (akin to manufacturing process development) and procedure execution (product production). (Note: Administrative work is part of lab operations but not an immediate concern here.) As such, we have to address the fact that lab work is part original science and part production work based on that science, e.g., as seen with quality control, clinical chemistry, and high-throughput screening labs. The routine production work of these and other labs can benefit most from <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_automation\" title=\"Laboratory automation\" class=\"wiki-link\" data-key=\"0061880849aeaca05f8aa27ae171f331\">automation<\/a> efforts. We need to think more broadly about the use of automation technologies\u2014driving their development\u2014instead of waiting to see what vendors develop. \n<\/p><p>Where manufacturing and lab work differ is in the scale of the work environment, the nature of the work station equipment, the skills needed to carry out the work, and the adaptability of those doing the work to unexpected situations.\n<\/p><p>My hope is that this guide will get laboratory managers and other stakeholders to begin thinking more about planning and technology management, as well as the need for more education in that work.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Trends_in_science_applications\">Trends in science applications<\/span><\/h4>\n<p>If new science isn\u2019t being developed, vendors will add digital hardware and software technology to existing equipment to improve capabilities and ease-of-use, separating themselves from the competition. However, there is still an obvious need for an independent organization to evaluate that technology (i.e., the lab version of <i>Consumer Reports<\/i>); as is, that evaluation process, done properly, would be time consuming for individual labs and would require a consistent methodology. With the increased use of automation, we need to do this better, such that the results can be used more widely (rather than every lab doing their own thing) and with more flexibility, using specialized equipment designed for automation applications.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/Artificial_intelligence\" title=\"Artificial intelligence\" class=\"wiki-link\" data-key=\"0c45a597361ca47e1cd8112af676276e\">Artificial intelligence<\/a> (AI) and <a href=\"https:\/\/www.limswiki.org\/index.php\/Machine_learning\" title=\"Machine learning\" class=\"wiki-link\" data-key=\"79aab39cfa124c958cd1dbcab3dde122\">machine learning<\/a> (ML) are two other trending topics, but they are not quite ready for widespread real-world applications. However, modern examples still exist:\n<\/p>\n<ul><li> Having a system that can bring up all relevant information on a research question\u2014a sort of super Google\u2014or a variation of IBM\u2019s Watson could have significant benefits.<\/li>\n<li> Analyzing complex data or large volumes of data could be beneficial, e.g., the analysis of radio astronomy data to find fast radio bursts (FRB).<sup id=\"rdp-ebb-cite_ref-HsuIsIt18_5-0\" class=\"reference\"><a href=\"#cite_note-HsuIsIt18-5\">[3]<\/a><\/sup><\/li>\n<li> \"[A] team at Glasgow University has paired a machine-learning system with a robot that can run and analyze its own chemical reaction. The result is a system that can figure out every reaction that's possible from a given set of starting materials.\"<sup id=\"rdp-ebb-cite_ref-TimmerAIPlus18_6-0\" class=\"reference\"><a href=\"#cite_note-TimmerAIPlus18-6\">[4]<\/a><\/sup><\/li>\n<li> HelixAI is using Amazon's Alexa as a digital assitant for laboratory work.<sup id=\"rdp-ebb-cite_ref-HelixAIHome_7-0\" class=\"reference\"><a href=\"#cite_note-HelixAIHome-7\">[5]<\/a><\/sup><\/li><\/ul>\n<p>However there are problems using these technologies. ML systems have been shown to be susceptible to biases in their output depending on the nature and quality of the training materials. As for AI, at least in the public domain, we really don\u2019t know what that is, and what we think it is keeps changing as purported example emerge. One large problem for lab use is whether or not you can trust the results of an AI's output. We are used to the idea that lab systems and methods have to be validated before they are trusted, so how do you validate a system based on ML or AI?\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Education\">Education<\/span><\/h3>\n<p>The major issue in all of this is having people educated to the point where they can successfully handle the planning and management of laboratory technology. One key point: most lab management programs focus on personnel issues, but managers also have to understand the capabilities and limitations of information technology and automation systems.\n<\/p><p>One result of the <a href=\"https:\/\/www.limswiki.org\/index.php\/COVID-19\" title=\"COVID-19\" class=\"mw-redirect wiki-link\" data-key=\"da9bd20c492b2a17074ad66c2fe25652\">COVID-19<\/a> pandemic is that we are seeing the limitations of the four-year undergraduate degree program in science and engineering, as well as the state of remote learning. With the addition of information technologies, general systems thinking and modeling<sup id=\"rdp-ebb-cite_ref-8\" class=\"reference\"><a href=\"#cite_note-8\">[c]<\/a><\/sup>, statistical experimental design, and statistical process control have become multidisciplinary fields. We need options for continuing education throughout people\u2019s careers so they can maintain their competence and learn new material as needed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Making_laboratory_informatics_and_automation_work\">Making laboratory informatics and automation work<\/span><\/h2>\n<p>Making <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_informatics\" title=\"Laboratory informatics\" class=\"wiki-link\" data-key=\"00edfa43edcde538a695f6d429280301\">laboratory informatics<\/a> and automation work? \"Isn\u2019t that a job for IT or lab personnel?\" someone might ask. One of the problems in modern science is the development of specialists in disciplines. The laboratory and IT fields have many specialties, and specialists can be very good within those areas while at the same time not having an appreciation of wider operational issues. Topics like lab operations, technology management, and planning aren\u2019t covered in formal education courses, and they're often not well-covered in short courses or online programs.\n<\/p><p>\u201cMaking it work\u201d depends on planning performed at a high enough level in the organization to encompass all affected facilities and departments, including information technology (IT) and facilities management. This wider perspective gives us the potential for synergistic operations across labs, consistent policies for facilities management and IT, and more effective use of outside resources (e.g., lab information technology support staff [LAB-IT], laboratory automation engineers [LAEs]<sup id=\"rdp-ebb-cite_ref-9\" class=\"reference\"><a href=\"#cite_note-9\">[d]<\/a><\/sup>, equipment vendors, etc.). \n<\/p><p>We need to apply the same diligence to planning lab operations as we do any other critical corporate resource. Planning provides a structure for enabling effective and successful lab operations.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Introduction_to_this_section\">Introduction to this section<\/span><\/h3>\n<p>The common view of science laboratories is that of rooms filled with glassware, lab benches, and instruments being used by scientists to carry out experiments. While this is a reasonable perspective, what isn\u2019t as visually obvious is the end result of that work: the development of knowledge, information, and data.\n<\/p><p>The progress of laboratory work\u2014as well as the planning, documentation, analytical results related to that work\u2014have been recorded in paper-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_notebook\" title=\"Laboratory notebook\" class=\"wiki-link\" data-key=\"be60c7be96aba8e9a84537fd8835fa54\">laboratory notebooks<\/a> for generations, and people are still using them today. However, these aren't the only paper records that have existed and are still in use; scientists also depend on charts, log books, administrative records, reports, indexes, and reference material. The latter half of the twentieth century introduced electronics into the lab and with it electronic recording in the form of computers and data storage systems. Early adopters of these technologies had to extend their expertise into the information technology realm because there were few people who understood both these new devices and their application to lab work\u2014you had to be an expert in both laboratory science and computer science.\n<\/p><p>In the 1980s and 90s, computers became commonplace and where once you had to understand hardware, software, operating systems, programming and application packages, you then simply had to know how to turn them on; no more impressive arrays of blinking lights, just a blinking cursor waiting for you to do something.\n<\/p><p>As systems gained ease-of-use, however, we lost the basic understanding of what these systems were and what they did, that they had faults, and that if we didn\u2019t plan for their effective use and counter those faults, we were opening ourselves to unpleasant surprises. The consequences at times were system crashes, lost data, and a lack of a real understanding of how the output of an instrument was transformed into a set of numbers, which meant we couldn\u2019t completely account for the results we were reporting. \n<\/p><p>We need to step back, take control, and institute effective technology planning and management, with appropriate corresponding education, so that the various data we are putting into laboratory informatics technologies have the desired outcome. We need to ensure that these technologies are providing a foundation for improving laboratory operations efficiency and a solid return on investment (ROI), while substantively advancing your business' ability to work and be productive. That's the purpose of the work we'll be discussing.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"The_point_of_planning\">The point of planning<\/span><\/h4>\n<p>The point of planning and technology management is pretty simple: to ensure ...\n<\/p>\n<ul><li> that the right technologies are in people's hands when they need them, and<\/li>\n<li> that those technologies complement each other as much as possible.<\/li><\/ul>\n<p>These are straightforward statements with a lot packed into them.\n<\/p><p>Regarding the first point, the key words are \u201cthe right technologies.\u201d In order to define what that means, lab personnel have to understand the technologies in question and how they apply to their work. If those personnel have used or were taught about the technologies under consideration, it should be easy enough to do. However, laboratory informatics doesn\u2019t fall into that basket of things. The level of understanding has to be more than superficial. While personnel don\u2019t have to be software developers, they do have to understand what is happening within informatics systems, and how data processing handles their data and produces results. Determining the \u201cright technologies\u201d depends on the quality and depth of education possessed by lab personnel, and eventually by lab information technology support staff (LAB-IT?) as they become involved in the selection process.\n<\/p><p>The second point also has a lot buried inside it. Lab managers and personnel are used to specifying and purchasing items (e.g., instruments) as discrete tools. When it comes to laboratory informatics, we\u2019re working with things that connect to each other, in addition to performing a task. When we explore those connections, we need to assess how they are made, what we expect to gain, what compatibility issues exist, how to support them, how to upgrade them, what their life cycle is, etc. Most of the inter-connected devices people encounter in their daily lives are things that were expected to be connected with using a limited set of choices; the vendors know what those choices are and make it easy to do so, or otherwise their products won\u2019t sell. The laboratory technology market, on the other hand, is too open-ended. The options for physical connections might be there, but are they the right ones, and will they work? Do you have a good relationship with your IT people, and are they able to help (not a given)? Again, education is a major factor.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Who_is_responsible_for_laboratory_technology_planning_and_management_.28TPM.29.3F\">Who is responsible for laboratory technology planning and management (TPM)?<\/span><\/h4>\n<p>When asking who is responsible for TPM, the question really is \"who are the TPM stakeholders,\" or \"who has an invested interest in seeing TPM prove successful?\"\n<\/p>\n<ul><li> Corporate or organizational management: These stakeholders set priorities and authorize funding, while also rationalizing and coordinating goals between groups. Unless the organization has a strong scientific base, they may not appreciate the options and benefits of TPM in lab work, or the possibilities of connecting the lab into the rest of the corporate data structure.<\/li>\n<li> Laboratory management: These stakeholders are responsible for developing and implementing plans, as well as translating corporate goals into lab priorities.<\/li>\n<li> Laboratory personnel: These stakeholders are the ones that actually do the work. However, they are in the best position to understand where technologies can be applied. They would also be relied on to provide user requirements documents for new projects and meet both internal and external (e.g., <a href=\"https:\/\/www.limswiki.org\/index.php\/Food_and_Drug_Administration\" title=\"Food and Drug Administration\" class=\"wiki-link\" data-key=\"e2be8927071ac419c0929f7aa1ede7fe\">Food and Drug Administration<\/a> [FDA], <a href=\"https:\/\/www.limswiki.org\/index.php\/United_States_Environmental_Protection_Agency\" title=\"United States Environmental Protection Agency\" class=\"wiki-link\" data-key=\"877b052e12328aa52f6f7c3f2d56f99a\">Environmental Protection Agency<\/a> [EPA], <a href=\"https:\/\/www.limswiki.org\/index.php\/International_Organization_for_Standardization\" title=\"International Organization for Standardization\" class=\"wiki-link\" data-key=\"116defc5d89c8a55f5b7c1be0790b442\">International Organization for Standardization<\/a> [ISO], etc.) performance guidelines.<\/li>\n<li> IT management and their support staff: While these stakeholders' traditional role is the support of computers, connected devices (e.g., printers, etc.) and network infrastructure, they may also be the first line of support for computers connected to lab equipment. IT staff either need to be educated to meet that need and support lab personnel, or have additional resources available to them. They may also be asked to participate in planning activities as subject matter experts on computing hardware and software.<\/li>\n<li> LAB-IT specialists: These stakeholders act as the \"additional resources\" alluded to in the previous point. These are crossover specialists that span the lab and IT spaces and can provide informed support to both. In most organizations, aside from large science-based companies, this isn\u2019t a real \"position,\" although once stated, its role is immediately recognized. In the past, I\u2019ve also referenced these stakeholders as being \u201claboratory automation engineers.\u201d<sup id=\"rdp-ebb-cite_ref-LiscouskiAreYou06_10-0\" class=\"reference\"><a href=\"#cite_note-LiscouskiAreYou06-10\">[6]<\/a><\/sup><\/li>\n<li> Facility management: These stakeholders need to ensure that the facilities support the evolving state of laboratory workspace requirements as traditional formats change to support robotics, instrumentation, computers, material flow, power, and HVAC requirements.<\/li><\/ul>\n<p>Carrying out this work is going to rely heavily on expanding the education of those participating in the planning work; the subject matter goes well beyond material covered in degree programs.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Why_put_so_much_effort_into_planning_and_technology_management.3F\">Why put so much effort into planning and technology management?<\/span><\/h4>\n<p>Earlier we mentioned paper laboratory notebooks, the most common recording device since scientific research began (although for sheer volume, it may have been eclipsed by computer hard drives). Have you ever wondered about the economics of laboratory notebooks? Cost is easy to understand, but the value of the data and information that is recorded there requires further explanation.\n<\/p><p>The value of the material recorded in a notebook depends on two key factors: the quality of the work and an inherent ability to put that documented work to use. The quality of the work is a function of those doing the work, how diligent they are, and the veracity of what has been written down. The inherent ability to use it depends upon the clarity of the writing, people\u2019s ability to understand it without recourse to the author, and access to the material. That last point is extremely important. Just by glancing at Figure 1, you can figure out where this is going.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"18969a8a26ae94a6de9754c2cd6aa45e\"><img alt=\"Fig1 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f1\/Fig1_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Paper notebooks' cost vs. value, as a function of usage<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As a scientist\u2019s notebook fills with entries, it gains value because of the content. Once filled, it reaches an upper limit and is placed in a library. There it takes a slight drop in value because its ease-of-access has changed; it isn\u2019t readily at hand. As library space fills, the notebooks are moved to secondary storage (in one company I worked at, secondary storage consisted of trailers in a parking lot). Costs go up due to the cost of owning or renting the secondary storage and the space they take. The object's value drops, not because of the content but due to the difficulty in retrieving that content (e.g., which trailer? which box?). Unless the project is still active, the normal turn-over of personnel (e.g., via promotions, movement around the company, leaving the company) mean that institutional memory diminishes and people begin to forget the work exists. If few researchers can remember it, find it, and access it, the value drops regardless of the resources that went into the work. That is compounded by the potential for physical deterioration of the object (e.g., water damage, mice, etc.).\n<\/p><p>Preventing the loss of access to the results of your investment in R&D projects will rely on information technology. That reliance will be built upon planning an effective informatics environment, which is precisely where this discussion is going. How is putting you lab results into a computer system any different than a paper-based laboratory notebook? There are obvious things like faster searching and so on, but from our previous discussion on them, not much is different; you still have essentially a single point of failure, unless you plan for that eventuality. That is the fundamental difference and what will drive the rest of this writing: \n<\/p>\n<dl><dd>Planning builds in reliability, security, and protection against loss. (Oh, and it allows us to work better, too!)<\/dd><\/dl>\n<p>You could plan for failure in a paper-based system by making copies, but those copies still represent paper that has to be physically managed. With electronic systems, we can plan for failure by using automated <a href=\"https:\/\/www.limswiki.org\/index.php\/Backup\" title=\"Backup\" class=\"wiki-link\" data-key=\"e12548e6bf5f28bfee99099fe8662dde\">backup<\/a> procedures that make faithful copies, as many as we\u2019d like, at low cost. This issue isn\u2019t unique to laboratory notebooks, but it is a problem for organizations that depends on paper records.\n<\/p><p>The difference between writing on paper and using electronic systems isn\u2019t limited to how the document is realized. If you were to use a typewriter, the characters would show up on the paper and you'd be able to read them; all you needed was the ability to read (which could include braille formats) and understand what was written. However, if you were using a word processor, the keystrokes would be captured by software, displayed on the screen, placed in the computer\u2019s memory, and then written to storage. If you want to read the file, you need something\u2014software\u2014to retrieve it from storage, interpret the file contents, determine how to display it, and then display it. Without that software the file is useless. A complete backup process has to include the software needed to read the file, plus all the underlying components that it depends upon. You could correctly argue that the hardware is required as well, but there are economic tradeoffs as well as practical ones; you could transfer the file to other hardware and read it there for example. \n<\/p><p>That point brings us to the second subject of this writing: technology management. What do I have to do to make sure that I have the right tools to enable me to work? The problem is simple enough when all you're concerned with is writing and preparing accompanying graphics. Upon shifting the conversation to laboratory computing, it gets more complicated. Rather than being concerned with one computer and a few software packages, you have computers that acquire and process data in real-time<sup id=\"rdp-ebb-cite_ref-11\" class=\"reference\"><a href=\"#cite_note-11\">[e]<\/a><\/sup>, transmit it to other computers for storage in databases, and systems that control <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">sample<\/a> processing and administrative work. Not only do the individual computer systems and the equipment and people they support have to work well, but also they have to work cooperatively, and that is why we have to address planning and technology management in laboratory work.\n<\/p><p>That brings us to a consideration of what lab work is all about.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Different_ways_of_looking_at_laboratories\">Different ways of looking at laboratories<\/span><\/h2>\n<p>When you think about a \u201claboratory,\u201d a lot depends on your perspective: are you on the outside looking in, do you work in a lab, or are you taking that high school chemistry class? When someone walks into a science laboratory, the initial impression is that of confusing collection of stuff, unless they're familiar with the setting. \u201cStuff\u201d can consist of instruments, glassware, tubing, robots, incubators, refrigerators and freezers, and even petri dishes, cages, fish tanks, and more depending on the kind of work that is being pursued.\n<\/p><p>From a corporate point of view, a \"laboratory\" can appear differently and have different functions. Possible corporate views of the laboratory include:\n<\/p>\n<ol><li> A laboratory is where questions are studied, which may support other projects or provide a source of new products, acting as basic and applied R&D. What is expected out of these labs is the development of new knowledge, usually in the form of reports or other documentation that can move a project forward.<\/li>\n<li> A laboratory acts as a research testing facility (e.g., analytical, physical properties, mechanical, electronics, etc.) that supports research and manufacturing through the development of new test methods, special analysis projects, troubleshooting techniques, and both routine and non-routine testing. The laboratory's results come in the form of reports, test procedures, and other types of documented information.<\/li>\n<li> A laboratory acts as a <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_assurance\" title=\"Quality assurance\" class=\"wiki-link\" data-key=\"2ede4490f0ea707b14456f44439c0984\">quality assurance<\/a>\/<a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> (QA\/QC) facility that provides routine testing, producing information in support of production facilities. This can include incoming materials testing, product testing, and product certification.<\/li><\/ol>\n<p>Typically, stakeholders outside the lab are looking for some form of result that can be used to move projects and other work forward. They want it done quickly and at low cost, but also want the work to be of high quality and reliability. Those considerations help set the goals for lab operations.\n<\/p><p>Within the laboratory there are two basic operating modes or <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a>: project-driven or task-driven work. With project-driven workflows, a project goal is set, experiments are planned and carried out, the results are evaluated, and a follow-up course of action is determined. This all requires careful documentation for the planning and execution of lab work. This can also include developing and revising standard operating procedures (SOPs). Task-driven workflows, on the other hand, essentially depend on the specific steps of a process. A collection of samples needs to be processed according to an SOP, and the results recorded. Depending upon the nature of the SOP and the number of samples that have to be processed, the work can be done manually, using instruments, or with partial or full automation, including robotics. With the exception of QA\/QC labs, a given laboratory can use a combination of these modes or workflows over time as work progresses and the internal\/external resources become available. QA\/QC labs are almost exclusively task-driven; contract testing labs are as well, although they may take on project-driven work.\n<\/p><p>Within the realm of laboratory informatics, project-focused work centers on the <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_laboratory_notebook\" title=\"Electronic laboratory notebook\" class=\"wiki-link\" data-key=\"a9fbbd5e0807980106763fab31f1e72f\">electronic laboratory notebook<\/a> (ELN), which can be described as a lab-wide diary of work and results. Task-driven work is organized around the <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management system<\/a> (LIMS)\u2014or <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS) in clinical lab settings\u2014which can be viewed as a workflow manager of tests to be done, results to be recorded, and analyses to be finalized. Both of these technologies replaced the paper-based laboratory notebook discussed earlier, coming with considerable improvements in productivity. And although ELNs are considerably more expensive than paper systems, the short- and long-term benefits of an ELN overshadow that cost issue.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Labs_in_transition.2C_from_manual_operation_to_modern_facilities\">Labs in transition, from manual operation to modern facilities<\/span><\/h2>\n<div class=\"thumb tright\"><div class=\"thumbinner\" style=\"width:502px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:PSM_V43_D075_Chemical_laboratory.jpg\" class=\"image wiki-link\" data-key=\"f30aee78def64774a6987a368702b555\"><img alt=\"\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/b\/ba\/PSM_V43_D075_Chemical_laboratory.jpg\" class=\"thumbimage\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a> <div class=\"thumbcaption\"><div class=\"magnify\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:PSM_V43_D075_Chemical_laboratory.jpg\" class=\"internal wiki-link\" title=\"Enlarge\" data-key=\"f30aee78def64774a6987a368702b555\"><\/a><\/div><b>Figure 2.<\/b> Chemical laboratory at Oswego SUNY 1893<\/div><\/div><\/div>Laboratories didn\u2019t start with lots of electronic components; they began with people, lab benches, glassware, Bunsen burners, and other equipment. Lab operations were primarily concerned with peoples' ability to work. The technology was fairly simple by today\u2019s standards (Figure 2), and an individual\u2019s skills were the driving factor in producing quality results.\n<p>For the most part, the skills you learned in school were the skills you needed to be successful here as far as technical matters went; management education was another issue. That changed when electronic instrumentation became available. Analog instruments such as scanning <a href=\"https:\/\/www.limswiki.org\/index.php\/Spectrophotometer\" title=\"Spectrophotometer\" class=\"wiki-link\" data-key=\"6382bb48c914f3c490400c13f9eb16e6\">spectrophotometers<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Chromatography\" title=\"Chromatography\" class=\"wiki-link\" data-key=\"2615535d1f14c6cffdfad7285999ad9d\">chromatographs<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Mass_spectrometry\" title=\"Mass spectrometry\" class=\"wiki-link\" data-key=\"fb548eafe2596c35d7ea741849aa83d4\">mass spectrometer<\/a>, differential scanning <a href=\"https:\/\/www.limswiki.org\/index.php\/Calorimeter\" title=\"Calorimeter\" class=\"wiki-link\" data-key=\"af2c641f435c259a996f13fc0c612eed\">calorimeters<\/a>, tensile testers, and so on introduced a new career path to laboratory work: the instrument specialist, who combined an understanding of the basic science with the an understanding of the instrument\u2019s design, as well as how to use it (and modify it where needed), maintain it, troubleshoot issues, and analyze the results. Specialization created a problem for schools: they couldn\u2019t afford all the equipment, find knowledgeable instructors, and encourage room in the curriculum for the expanding subject matter. Schools were no longer able to educate people to meet the requirements of industry and graduate-level academia. And then digital electronics happened. Computers first became attached to instruments, and then incorporated into the instrumentation.<sup id=\"rdp-ebb-cite_ref-12\" class=\"reference\"><a href=\"#cite_note-12\">[f]<\/a><\/sup>\n<\/p><p>The addition of computer hardware and software to an instrument increased the depth of specialization in those techniques. Not only did you have to understand the science noted above, but also the use of computer programs used to work with the instrument, how to collect the data, and how to perform the analysis. An entire new layer of skills was added to an already complex subject.\n<\/p><p>The latest level of complexity added to laboratory operations has been the incorporation of LIMS, ELNs, <a href=\"https:\/\/www.limswiki.org\/index.php\/Scientific_data_management_system\" title=\"Scientific data management system\" class=\"wiki-link\" data-key=\"9f38d322b743f578fef487b6f3d7c253\">scientific data management systems<\/a> (SDMS), and <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_execution_system\" title=\"Laboratory execution system\" class=\"wiki-link\" data-key=\"774bdcab852f4d09565f0486bfafc26a\">laboratory execution systems<\/a> (LES) either as stand-alone modules or combined into more integrated packages or \"platforms.\"\n<\/p>\n<h3><span class=\"mw-headline\" id=\"There.27s_a_plan_for_that.3F\">There's a plan for that?<\/span><\/h3>\n<p>It is rare to find a lab that has an informatics plan or strategy in place before the first computer comes through the door; those machines enter as part of an instrument-computer control system. Several computers may use that route to become part of the lab's technology base before people realize that they need to start taking lab computing seriously, including how to handle backups, maintenance, support, etc.\n<\/p><p>First computers come into the lab, and then the planning begins, often months later, as an incremental planning effort, which is the complete reverse of how things need to be developed. Planning is essential as soon as you decide that a lab space will be created. That almost never happens, in part because no one has told you that is required, let alone why or how to go about it.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Thinking_about_a_model_for_lab_operations\">Thinking about a model for lab operations<\/span><\/h3>\n<p>The basic purpose of laboratory work is to answer questions. \u201cHow do we make this work?\u201d \u201cWhat is it?\u201d \u201cWhat\u2019s the purity of this material?\u201d These questions and others like them occur in chemistry, physics, and the biological sciences. Answering those questions is a matter of gathering data and information through observation and experimental work, organizing it, analyzing it, and determining the next steps needed as the work moves forward (Figure 3). Effective organization is essential, as lab personnel will need to search data and information, extract it, move it from one data system to another for analysis, make decisions, update planning, and produce interim and ultimately final reports.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"45986849e53d67a4c64847724f493759\"><img alt=\"Fig3 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/56\/Fig3_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Simplified flow of data\/information from sources to collection system<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Once the planning is done, scientific work generally begins with collecting observations and measurements (Data\/Information Sources 1\u20134, Figure 3) from a variety of sources. Lab bench work usually involves instrumentation, and many instruments have computer controls and data systems as part of them. This is the more visible part of lab work and the one that matches people\u2019s expectations for a \u201cscientific lab.\u201d This is where most of the money is spent on equipment, materials, and people\u2019s expertise and time. All that expenditure of resources results in \u201cthe pH of the glowing liquid is 6.5,\u201d \u201cthe concentration of iron in the icky stuff is 1500 ppm,\u201d and so on. That\u2019s the end result of all those resources, time, and effort put into the scientific workflow. That\u2019s why you built a million-dollar facility (in some spheres of science such as astronomy, high energy physics, and the space sciences, the cost of collection is significantly higher). So what do you do with those results? Prior to the 1970s, the collection points were paper: forms, notebooks, and other document, all with their earlier discussed issues.\n<\/p><p>The material on those instrument data systems needs to be moved to an intermediate system for long-term storage and reference (the second step of Figure 3). This is needed because those initial data systems may fail, be replaced, or added to as the work continues. After all, the data and information they\u2019ve collected needs to be preserved, organized, and managed to support continued lab work.\n<\/p><p>The analyzed results need to be collected into a reference system that is the basis of long-term analysis, management\/administration work, and reporting. This last system in the flow is the central hub of lab activities; it is also the distribution point for material sent to other parts of the organization (the third and fourth stages of Figure 3). While it is natural for scientists to focus on the production of data and information, the organization and centralized management of the results of laboratory work needs to be a primary consideration. That organization will be focused of short- and long-term <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">data analysis<\/a> and evaluation. The results of this get used to demonstrate the lab's performance towards meeting its goals, and it will show those investing in your work that you\u2019ve got your management act together, which is useful when looking for continued support.\n<\/p><p>Today, those systems come in two basic forms: LIMS and ELN. The details of those systems are the subject of a number of articles and books.<sup id=\"rdp-ebb-cite_ref-LiscouskiWhich15_13-0\" class=\"reference\"><a href=\"#cite_note-LiscouskiWhich15-13\">[7]<\/a><\/sup> Without getting into too much detail:\n<\/p>\n<ul><li> LIMS are used to support testing labs managing sample workflows and planning, as well as cataloging results (e.g., short text and numerical information).<\/li><\/ul>\n<ul><li> ELNs are usually found in research functioning as an electronic diary of lab work for one or more scientists and technicians. The entries may contain extensive textural material, numerical entries, charts, graphics, etc. The ELN is generally more flexible than a LIMS.<\/li><\/ul>\n<p>That distinction is simplistic; some labs support both activities and need both types of systems, or even a hybrid package. However, the description is sufficient to get us to the next point: the lifespan of systems varies, depending on where you are looking in Figure 3's model. Figure 4 gives a comparison.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"71a82219dd4e0aba35959d868be597e3\"><img alt=\"Fig4 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/ad\/Fig4_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> The relative lifespan of laboratory systems<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The experimental methods\/procedures used in lab work will change over time as the needs of the lab change. Older instruments may be updated and new ones introduced. Retirement is a problem, particularly if data systems are part of the equipment. You have to have access to the data. That need will live on long past the equipment's life. That is one reason that moving data and information to an intermediate system like an SDMS is important. However, in some circumstances, even that isn\u2019t going to be sufficient (regulated industries where the original data structures and software that generated them need to be preserved as an operating entity). In those cases, you may have old computers stacked up just in case you need access to their contents. A better way is to virtualize the systems as containers on servers that support a virtualized environment.\n<\/p><p>Virtualization\u2014making an electronic copy of computer system and running on a server\u2014is potentially a useful technology in lab work; while it won\u2019t participate in day-to-day activities it does have a role. Suppose you have an instrument-data system that is being replaced or retired. Maybe the computer is showing signs of aging or failing. What do you do with the files and software that are on the computer portion of the combination? You can\u2019t dispose of them because you may need access to those data files and software later. On the other hand, do you really want to collect computer systems that have to be maintained just to have access to the data if and when you need it? Instead, virtualization is a software\/hardware technology that allows you to make a complete copy of everything that is on that computer\u2014including operating system files, applications, and data files\u2014and stores it in one big file referred to as a \u201ccontainer.\u201d That container can be moved to a computer that is a virtual server and has software that emulates various operating environment, allowing the software in the container to run as if it were on its own computer hardware. A virtual server can support a lot of containers, and the operating systems in those containers can be updated as needed. The basic idea is that you don\u2019t need access to a separate physical computer; you just need the ability to run the software that was on it. If your reaction to that is one of dismay and confusion, it\u2019s time to buy your favorite IT person a cup of coffee and have a long talk. We\u2019ll get into more details when we cover data backup issues.\n<\/p><p><b>Why is this important to you?<\/b>\n<\/p><p>While the science behind producing results is the primary reason your lab exists, gaining the most value from the results is essential to the organization overall. That value is going to be governed by the quality of the results, ease of access, the ability to find and extract needed information easily, and a well-managed K\/I\/D architecture. All of that addresses a key point from management\u2019s perspective: return on investment or ROI. If you can demonstrate that your data systems are well organized and maintained, and that you can easily find and use the results from experimental work and contribute to advancing the organization\u2019s goals, you\u2019ll make it easier to demonstrate solid ROI and gain funding for projects, equipment, and people needed to meet your lab's goals.\n<\/p><p><br \/>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_seven_goals_of_planning_and_managing_lab_technologies\">The seven goals of planning and managing lab technologies<\/span><\/h2>\n<p>The preceding material described the need for planning and managing lab technologies, and making sure lab personnel are qualified and educated to participate in that work. The next step is the actual planning. There are at least two key aspects to that work: planning activities that are specific and unique to your lab(s) and addressing broader scope issues that are common to all labs. The discussion found in the rest of this guide is going to focus on the latter points.\n<\/p><p>Effective planning is accomplished by setting goals and determining how you are going to achieve them. The following sections of this guide look at those goals, specifically:\n<\/p>\n<ol><li> Supporting an environment that fosters productivity and innovation<\/li>\n<li> Developing high-quality data and information<\/li>\n<li> Managing knowledge, information, and data effectively, putting them in a structure that encourages use and protects value<\/li>\n<li> Ensuring a high level of data integrity at every step<\/li>\n<li> Addressing security throughout the lab <\/li>\n<li> Acquiring and developing \"products\" that support regulatory requirements<\/li>\n<li> Addressing systems integration and harmonization<\/li><\/ol>\n<p>The material below begins the sections on goal setting. Some of these goals are obvious and understandable, others like \u201charmonization\u201d are less so. The goals are provided as an introduction rather than an in-depth discussion. The intent is to offer something suitable for the purpose of this material and a basis for a more detailed exploration at a later point. The intent of these goals is not to tell you how to do things, but rather what things need to be addressed. The content is provided as a set of questions that you need to think about. The answers aren't mine to give, but rather yours to develop and implement; it's your lab. In many cases, developing and implementing those answers will be a joint effort by all stakeholders.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"First_goal:_Support_an_environment_that_fosters_productivity_and_innovation\">First goal: Support an environment that fosters productivity and innovation<\/span><\/h3>\n<p>In order to successfully plan for and manage lab technologies, the business environment should ideally be committed to fostering a work environment that encourages productivity and innovation. This requires:\n<\/p>\n<ul><li> proven, supportable workflow methodologies;<\/li>\n<li> educated personnel;<\/li>\n<li> fully functional, inter-departmental cooperation;<\/li>\n<li> management buy-in; and<\/li>\n<li> systems that meet users' needs.<\/li><\/ul>\n<p>This is one of those statements that people tend to read, say \u201csure,\u201d and move on. But before you do that, let\u2019s take a look at a few points. Innovation may be uniquely human (not even going to consider AI), and the ability to be \u201cinnovative\u201d may not be universal.\n<\/p><p>People need to be educated, be able to separate true facts from \u201cbeliefs,\u201d and question everything (which may require management support). Innovation doesn\u2019t happen in a highly structured environment, you need the freedom to question, challenge, etc. You also need the tools to work with. The inspiration that leads to innovation can happen anywhere, anytime. All of a sudden all the pieces fit. And then what? That is where a discussion of tools and this work come together.\n<\/p><p>If a sudden burst of inspiration hits, you want to do it now and not after traveling to an office, particularly if it is weekend or vacation. You need access to knowledge (e.g., documents, reports), information, and data (K\/I\/D). In order to do that, a few things have to be in place:\n<\/p>\n<ul><li> Business and operations K\/I\/D must be accessible.<\/li>\n<li> Systems security has to be such that a qualified user can gain access to K\/I\/D remotely, while preventing its unauthorized use.<\/li>\n<li> Qualified users must have the hardware and software tools required to access the K\/I\/D, work with it, and transmit the results of that work to whoever needs to see it.<\/li>\n<li> Qualified users must also be able to remotely initiate actions such as testing.<\/li><\/ul>\n<p>Those elements depend on a well-designed laboratory and corporate informatics infrastructure. Laboratory infrastructure is important because that is where the systems are that people need access to, and corporate infrastructure is important since corporate facilities have to provide access, controls, and security. Implementation of those corporate components has to be carefully thought through; they must be strong enough to frustrate unwarranted access (e.g., multi-factor logins) while allowing people to get real work done.\n<\/p><p>All of this requires flexibility and trust in people, an important part of corporate culture. This will become more important as society adjusts to new modes of working (e.g., working online due to a pandemic) and the realization that the fixed format work week isn\u2019t the only way people can be productive. For example, working from home or off-site is increasingly commonplace. Laboratory professionals work in two modes: intellectual, which can be done anywhere, and the lab bench, where physical research tasks are performed. We need to strike a balance between those modes and the need for in-person vs virtual contact.\n<\/p><p>Let's take another look at the previous Figure 3, which offered one possible structure for organizing lab systems:\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"45986849e53d67a4c64847724f493759\"><img alt=\"Fig3 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/56\/Fig3_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Simplified flow of data\/information from sources to collection system<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>This use of an intermediate file storage system like an SDMS and the aggregation of some instruments to a common computer (e.g., one chromatographic data system for all chromatographs vs. one per instrument) becomes more important for two reasons: 1. it limits the number of systems that have to be accessed to search, organize, extract, and work with K\/D\/I, and 2. it makes it easier to address security concerns. There are additional reasons why this organization of lab systems is advantageous, but we\u2019ll cover those in later installments. The critical point here is a sound informatics architecture is key to supporting innovation. People need access to tools and K\/D\/I when they are working, regardless of where they are working from. As such, those same people need to be well-versed in the capabilities of the systems available to them, how to access them, use them, and how to recognize \u201cmissing technologies,\u201d capabilities they need but don\u2019t have access to or simply don't exist.\n<\/p><p>Imagine this. A technology expert consults for two large organizations, one tightly controlled (Company A), the other with a liberal view of trusting people to do good work (Company B). In the first case, getting work done can be difficult, with the expert fighting through numerous reviews, sign-offs, and politics. Company A has a stated philosophy that they don\u2019t want to be the first in the market with a new product, but would rather be a strong number two. They justify their position through the cost of developing markets for new products: let someone else do the heavy lifting and follow behind them. This is not a culture that spawns innovation. Company B, however, thrives on innovation. While processes and procedures are certainly in place, the company has a more relaxed philosophy about work assignments. If the expert has a realizable idea, Company B lets them run with it, as long as they complete their assigned workload in a timely fashion. This is what spurs the human side of innovation.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Second_goal:_Develop_high-quality_data_and_information\">Second goal: Develop high-quality data and information<\/span><\/h3>\n<p>Asking staff to \"develop high-quality data and information\" seems like a pretty obvious point, but this is where professional experience and the rest of the world part company. Most of the world treats \u201cdata\u201d and \u201cinformation\u201d as interchangeable words. Not here.\n<\/p><p>There are three key words that are going to be important in this discussion of goals: knowledge, information, and data (K\/I\/D). We\u2019ll start with \u201cknowledge\u201d. The type of knowledge we will be looking at is at the laboratory\/corporate level, the stuff that governs how a laboratory operates, including reports, administrative material, and most importantly standard operating procedures (SOPs). SOPs tell us how lab work is carried out via its methods, procedures, etc. (This subject parallels the topic of \u201cdata integrity,\u201d which will be covered later.) Figure 5 positions K\/I\/D with respect to each other within laboratory processes.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"009d875ad276a207e3b7a3932e0d3bf1\"><img alt=\"Fig5 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/27\/Fig5_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Simplified flow of data\/information from sources to collection system<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The diagram in Figure 5 is a little complicated, and we\u2019ll get into the details as the material develops. For the moment, we\u2019ll concentrate on the elements in black.\n<\/p><p>As noted above, SOPs guide activities within the lab. As work is defined\u2014both research and testing\u2014SOPs have to be developed so that people know how to carry out their tasks consistently. Our first concern then is proper management of SOPs. Sounds simple, but in practice it isn\u2019t. It\u2019s a matter of first developing and updating the procedures, documenting them, and then managing both the documents and the lab personnel using them.\n<\/p><p>When developing, updating, and documenting procedures, a lab will primarily be looking at the science its working with and how regulatory requirements affect it, particularly in research environments. Once developed, those procedures will eventually need to be updated. But why is an update to a procedure needed? What will the effects of the update be based on the changes that were made, and how do the results of the new version compare to the previous version? That last point is important, and to answer it you need a reference sample that has been run repeatedly under the older version so that you have a solid history of the results (i.e., control chart) over time. You also need the ability to run that same reference sample under the new procedure to show that there are no differences, or that differences can be accounted for. If differences persist, what do you do about the previous test results under the old procedure?\n<\/p><p>The idea of running one or more stable reference samples periodically is a matter of instituting statistical process control over the analysis process. It can show that a process is under control, detect drift in results, and demonstrate that the lab is doing its job properly. If multiple analysts are doing the same work, it can also reveal how their work compares and if there are any problems. It is in effect looking over their shoulders, but that just comes with the job. If you find that the amount of reference material is running low, then phase in a replacement, running both samples in parallel to get a documented comparison with a clean transition from one reference sample to another. It\u2019s a lot of work and it\u2019s annoying, but you\u2019ll have a solid response when asked \u201care you confident in these results?\u201d You can then say, \u201cYes, and here is the evidence to back it up.\u201d\n<\/p><p>After the SOPs have been documented, they must then be effectively managed and implemented. First, take note of the education and experience required for lab personnel to properly implement any SOP. Periodic evaluation (or even certification) would be useful to ensure things are working as they should. This is particularly true of procedures that aren\u2019t run often, as people may forget things. \n<\/p><p>Another issue of concern with managing SOPs is how to manage versioning. Consider two labs. Lab 1 is a well-run lab. When a new procedure is issued, the lab secretary visits each analyst, takes their copy of\nthe old method, destroys it, provides a copy of the new one, requires the analyst sign for receipt, and later requires a second signature after the method has been reviewed and understood. Additional education is also provided on an as-needed basis. Lab 2 has good intentions, but it's not as proactive as Lab 1. Lab 2 retains all documents on a central server. Analysts are able to copy a method to their machines and use it. However, there is no formalized method of letting people know when a new method is released. At any given time there may be several analysts running the same method using different versions of the related SOP. The end result is having a mix of samples run by different people according to different SOPs. \n<\/p><p>This comparison of two labs isn\u2019t electronic versions vs. paper, but rather a formal management structure vs. a loose one. There\u2019s no problem maintaining SOPs in an electronic format, as there are many benefits, but there shouldn\u2019t be any question about the current version, and there should be a clear process for notifying people about updates while also ensuring that analysts are currently educated in the new method's use.\n<\/p><p>Managing this set of problems\u2014analyst education, versions of SOPs, qualification of equipment, current reagents, etc.\u2014 was the foundation for one of the early original ELNs, SmartLab by Velquest, now developed as a LES by <a href=\"https:\/\/www.limswiki.org\/index.php\/Dassault_Syst%C3%A8mes_SA\" title=\"Dassault Syst\u00e8mes SA\" class=\"wiki-link\" data-key=\"1be69bd73e35bc3db0c3229284bf9416\">Dassault Syst\u00e8mes<\/a> as part of the BIOVIA product line. And while Dassault's LES, and much of the Biovia product line, narrowly focuses on their intended market, the product remains suitable for any lab where careful control over procedure execution is warranted. This is important to note, as a LES is designed to guide a person through a procedure from start to finish, making it one step away from engaging in a full robotics system (robotics may play a role in stages of the process). The use of an LES doesn\u2019t mean that personnel aren\u2019t trusted or deemed incompetent; rather, it is a mechanism for developing documented evidence that methods have been executed correctly. That evidence builds confidence in results.\n<\/p><p>LESs are available from several vendors, often as part of their LIMS or ELN offerings. Using any of these systems requires planning and scripting (a gentler way of saying \u201cprogramming\u201d), and the cost of implementation has to be balanced against the need (does the execution of a method require that level of sophistication) and ROI.\n<\/p><p>Up to this point, we\u2019ve looked at developing and managing SOPs, as well as at least one means of controlling experiment\/procedure execution. However, there are other ways of going about this, including manual and full robotics systems. Figure 6 takes us farther down the K\/I\/D model to elaborate further on experiment\/procedure execution.<sup id=\"rdp-ebb-cite_ref-14\" class=\"reference\"><a href=\"#cite_note-14\">[g]<\/a><\/sup>\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"085c4095953e3d63bb21a90911219f3c\"><img alt=\"Fig6 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/13\/Fig6_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> K\/I\/D flow diagram focusing on experiment\/procedure execution<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As we move from knowledge development and management (i.e., SOPs), and then on to sample preparation (i.e., pre-experiment), the next step is usually some sort of measurement by an instrument, whether it is pH meter or spectrometer, yielding your result. That brings us to two words we noted earlier: \"data\" and \"information.\" We'll note the differences between the two using a <a href=\"https:\/\/www.limswiki.org\/index.php\/Gas_chromatography\" title=\"Gas chromatography\" class=\"wiki-link\" data-key=\"e621fc6f90266fbc8db27d516e9cbb94\">gas chromatography<\/a> system as an example (Figure 7), as it and other base chromatography systems are among the most widely used of upper-tier instrumentation and widely found in labs where chemical analysis is performed.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"1bf85a473790960af479c0df53f9c4b6\"><img alt=\"Fig7 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/67\/Fig7_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 7.<\/b> Gas chromatography \"data\" and \"information\"<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As we look at Figure 7, we notice to the right of the vertical blue line is an output signal from a gas chromatograph. This is what chromatographers analyzed and measured when they carried out their work. The addition of a computer made life easier by removing the burden of calculations, but it also added complexity to the work in the form of having to manage the captured electronic data and information. An analog-to-digital (A\/D) converter transformed those smooth curves to a sequence of numbers that are processed to yield parameters that described the peaks, which in turn were used to calculate the amount of substance in the sample. Everything up to that last calculation\u2014left of the vertical blue line\u2014is \u201cdata,\u201d a set of numerical values that, taken individually, have no meaning by themself. It is only when we combine it with other data sets that we can calculate a meaningful result, which gives us \u201cinformation.\u201d\n<\/p><p>The paragraph above describes two different types of data:\n<\/p><p>1. the digitized detector output or \"raw data,\" constituting a series of readings that could be plotted to show the instrument output; and\n<\/p><p>2. the processed digitized data that provides descriptors about the output, with those descriptors depending largely upon the nature of the instrument (in the case of chromatography, the descriptors would be peak height, retention time, uncorrected peak area, peak widths, etc.).\n<\/p><p>Both are useful and neither of them should be discarded; the fact that you have the descriptors doesn\u2019t mean you don\u2019t need the raw data. The descriptors are processed data that depends on user-provided parameters. Changing the parameter can change the processing and the values assigned to those descriptors. If there are accuracy concerns, you need the raw data as a backup. Since storage is cheap, there really isn\u2019t any reason to discard anything, ever. (And in some regulatory environments, keeping raw data is mandated for a period of time.)\n<\/p><p>If you want to study the data and how it was processed to yield a result, you need more data, specifically the reference samples (standards) used to evaluate each sample. An instrument file by itself is almost useless without the reference material run with that sample. Ideally, you\u2019d want a file that contains all the sample and reference data that was analyzed in one session. That might be a series of manual samples analyzed or an entire auto-sampler tray.\n<\/p><p>Everything we've discussed here positively contributes to developing high-quality data and information. When methods are proven and you have documented evidence that they were executed by properly educated personnel using qualified reagents and instruments, you then have the instrument data to support each sample result and any other information gleaned from that data.\n<\/p><p>You might wonder what laboratorians did before computers. They dealt with stacks of spectra, rolls of chromatograms, and laboratory notebooks, all on paper. If they wanted to find the data (e.g., a pen trace on paper) for a sample, they turned to the lab's physical filing system to locate it.<sup id=\"rdp-ebb-cite_ref-15\" class=\"reference\"><a href=\"#cite_note-15\">[h]<\/a><\/sup> Why does this matter? That has to do with our third goal.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Third_goal:_Manage_K.2FI.2FD_effectively.2C_putting_them_in_a_structure_that_encourages_use_and_protects_value\">Third goal: Manage K\/I\/D effectively, putting them in a structure that encourages use and protects value<\/span><\/h3>\n<p>In the previous section we introduced three key elements of laboratory work: knowledge, information, and data (K\/I\/D). Each of these is \u201cdatabase\u201d structures (\u201cdata\u201d in the general sense). We also looked at SOP management as an example of knowledge management, and distinguished \u201cdata\u201d and \u201cinformation\u201d management as separate but related concerns. We also introduced flow diagrams (Figures 5 and 6) that show the relationship and development of each of those elements.\n<\/p><p>In order for those elements to justify the cost of their development, they have to be placed in systems that encourage utilization and thus retain their value. Modern informatics tools assist in many ways:\n<\/p>\n<ul><li> <a href=\"https:\/\/www.limswiki.org\/index.php\/Document_management_system\" title=\"Document management system\" class=\"wiki-link\" data-key=\"b9d429b16b89b35fe74097a3e0c2d1ee\">Document management systems<\/a> support knowledge databases (and some LIMS and ELNs inherently support document management).<\/li>\n<li> LIMS and ELNs provide a solid base for laboratory information, and they may also support other administrative and operational functions.<\/li>\n<li> Instrument data systems and SDMS collect instrument output in the form of reports, data, and information.<\/li><\/ul>\n<p>You may notice there is significant functional redundancy as vendors try to create the \u201cultimate laboratory system.\u201d Part of lab management\u2019s responsibility is to define what the functional architecture should look like based on their current and perceived needs, rather than having it defined for them. It\u2019s a matter of knowing what is required and seeing what fits rather than fitting requirements into someone else\u2019s idea of what's needed.\n<\/p><p>Managing large database systems is only one aspect of handling K\/I\/D. Another aspect involves the consideration of <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud<\/a> vs. local storage systems. What option works best for your situation, is the easiest to manage, and is supported by IT? We also have to address the data held in various desktop and mobile computing devices, as well as bench top systems like instrument data systems. There are a number of considerations here, not the least of which is product turnover (e.g., new systems, retired systems, upgrades\/updates, etc.). (Some of these points will be covered latter on in other sections.)\n<\/p><p>What you should think about now is the number of computer systems and software packages that you use on a daily basis, some of which are connected to instruments. How many different vendors are involved? How big are vendors (e.g., small companies\/limit staff, large organizations)? How often do they upgrade their systems? What\u2019s the likelihood they\u2019ll be around in two or five years?\n<\/p><p>Also ask what data file formats the vendor uses; these formats vary widely among vendors. Some put everything in CSV files, others in proprietary formats. In the latter case, you may not be able to use the data files without the vendor's software. In order to maintain the ability to work with instrument data, you will have to manage the software needed to open files and work with it, in addition to just making sure you have copies of the data files. In short, if you have an instrument-computer combination that does some really nice stuff and you want to preserve the ability to gain value from that instrument's data files, you have to make a backup copy of the software environment and the data files. This is particularly important if you're considering retiring a system that you'll still want to access data from, plus you may have to maintain any underlying software license. This is where the previous conversation about virtualization and containers comes in.\n<\/p><p>If you think about a computer system it has two parts: hardware (e.g., circuit boards, hard drive, memory, etc.) and software (e.g., the OS, applications, data files, etc.). From the standpoint of the computer\u2019s processor, everything is either data or instructions read from one big file on the hard drive, which the operating system has segmented for housing different types of files (that segmentation is done for your convenience; the processor just sees it all as a source of instructions and data). Virtualization takes everything on the hard drive, turns it into a complete file, and places that file onto a virtualization server where it is stored as a file called a \u201ccontainer.\u201d That server allows you to log in, open a container, and run it as though it were still on the original computer. You may not be able to connect it the original instruments to the containerized environment, but all the data processing functions will still be there. As such, a collection of physical computers can become a collection of containers. An added benefit of virtualizations applies when you're worried about an upgrade creating havoc with your application; instead, make a container as a backup.<sup id=\"rdp-ebb-cite_ref-16\" class=\"reference\"><a href=\"#cite_note-16\">[i]<\/a><\/sup>\n<\/p><p>The advantage of all this is that you continue to have the ability to gain value and access to all of your data and information even if the original computer has gone to the recycle bin. This of course assumes your IT group supports virtualization servers, which provide an advantage in that they are easier to maintain and don\u2019t take up much space. In larger organization this may already be happening, and in smaller organizations a conversation may be had to determine IT's stance. The potential snag in all this is whether or not the software application's vendor license will cover the operation of their software on a virtual server. That is something you may want to negotiate as part of the purchase agreement when you buy the system.\n<\/p><p>This section has shown that effective management of K\/I\/D is more than just the typical consideration of database issues, system upgrades, and backups. You also have to maintain and support the entire operating system, the application, and the data file ecosystem so that you have both the files needed and the ability to work with them.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Fourth_goal:_Ensure_a_high_level_of_data_integrity_at_every_step\">Fourth goal: Ensure a high level of data integrity at every step<\/span><\/h3>\n<p>\u201cData integrity\u201d is an interesting couple of words. It shows up in marketing literature to get your attention, often because it's a significant regulatory concern. There are different aspects to the topic, and the attention given often depends on a vendor's product or the perspective of a particular author. In reality, it touches on all areas of laboratory work. The following is an introduction to the goal, with more detail given in later sections.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Definitions_of_data_integrity\">Definitions of data integrity<\/span><\/h4>\n<p>There are multiple definitions of \"data integrity.\" A broad encyclopedic definition can be found at Wikipedia, described as \"the maintenance of, and the assurance of, data accuracy and consistency over its entire life-cycle\" and \"a critical aspect to the design, implementation, and usage of any system that stores, processes, or retrieves data.\"<sup id=\"rdp-ebb-cite_ref-WPDataInt_17-0\" class=\"reference\"><a href=\"#cite_note-WPDataInt-17\">[8]<\/a><\/sup>\n<\/p><p>Another definition to consider is from a more regulatory perspective, that of the FDA. In their view, data integrity focuses on the completeness, consistency, accuracy, and validity of data, particularly through a mechanism called the ALCOA+ principles. This means the data should be<sup id=\"rdp-ebb-cite_ref-HarmonWhatIs20_18-0\" class=\"reference\"><a href=\"#cite_note-HarmonWhatIs20-18\">[9]<\/a><\/sup>:\n<\/p>\n<ul><li> <b>Attributable<\/b>: You can link the creation or alteration of data to the person responsible.<\/li>\n<li> <b>Legible<\/b>: The data can be read both visually and electronically.<\/li>\n<li> <b>Contemporaneous<\/b>: The data was created at the same time that the activity it relates to was conducted.<\/li>\n<li> <b>Original<\/b>: The source or primary documents relating to the activity the data records are available, or certified versions of those documents are available, e.g., a notebook or raw database. (This is one reason why you should collect and maintain as much data and information from an instrument as possible for each sample.)<\/li>\n<li> <b>Accurate<\/b>: The data is free of errors, and any amendments or edits are documented.<\/li><\/ul>\n<p>Plus, the data should be:\n<\/p>\n<ul><li> <b>Complete<\/b>: The data must include all related analyses, repeated results, and associated metadata.<\/li>\n<li> <b>Consistent<\/b>: The complete data record should maintain the full sequence of events, with date and time stamps, such that the steps can be repeated.<\/li>\n<li> <b>Enduring<\/b>: The data should be able to be retrieved throughout its intended or mandated lifetime.<\/li>\n<li> <b>Available<\/b>: The data is able to be accessed readily by authorized individuals when and where they need it.<\/li><\/ul>\n<p>Both definitions revolve around the same point: the data a lab produces has to be reliable. The term \"data integrity\" and its associated definitions are a bit misleading. If you read the paragraphs above you get the impression that the focus in on the results of laboratory work, when in fact it is about every aspect of laboratory work, including the methods used and those who conduct those methods.\n<\/p><p>In order to gain meaningful value from laboratory K\/I\/D, you have to be assured of its integrity; \u201cthe only thing worse than no data, is data you can\u2019t trust.\u201d That is the crux of the matter. How do you build that trust? Building a sense of confidence in a lab's data integrity efforts requires addressing three areas of concern and their paired intersections: science, people, and informatics technology. Once we have successfully managed those areas and intersection points, we are left with the intersection common to all of them: constructed confidence in a laboratory's data integrity efforts (Figure 8). \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"496854229785eef610b2b5f0620f7df4\"><img alt=\"Fig8 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/20\/Fig8_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 8.<\/b> The three areas contributing to data integrity and how they intersect<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<div class=\"thumb tright\"><div class=\"thumbinner\" style=\"width:152px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig9_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"fc1aa2b241f71809573f67794ab6e19f\"><img alt=\"\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3a\/Fig9_Liscouski_LabTechPlanMan20.png\" width=\"150\" height=\"146\" class=\"thumbimage\" \/><\/a> <div class=\"thumbcaption\"><div class=\"magnify\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig9_Liscouski_LabTechPlanMan20.png\" class=\"internal wiki-link\" title=\"Enlarge\" data-key=\"fc1aa2b241f71809573f67794ab6e19f\"><\/a><\/div><b>Figure 9.<\/b> The \"Science\" component<\/div><\/div><\/div>\n<p><b>The science<\/b>\n<\/p><p>We\u2019ll begin with a look at the scientific component of the conversation (Figure 9). Regardless of the kinds of questions being addressed, the process of answering them is rooted in methods and procedures. Within the context of this guide, those methods have to be validated or else your first step in building confidence has failed. If those methods end with electronic measurements, then that equipment (including settings, algorithms, analysis, and reporting) have to be fully understood and qualified for use in the validated process. The manufacturer's default settings should either be demonstrated as suitable or avoided.\n<\/p><p><br \/>\n<\/p><p><b>The people<\/b>\n<\/p>\n<div class=\"thumb tleft\"><div class=\"thumbinner\" style=\"width:152px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig10_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"71773f0cca9237f7694aadea8d9fe421\"><img alt=\"\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/62\/Fig10_Liscouski_LabTechPlanMan20.png\" width=\"150\" height=\"142\" class=\"thumbimage\" \/><\/a> <div class=\"thumbcaption\"><div class=\"magnify\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig10_Liscouski_LabTechPlanMan20.png\" class=\"internal wiki-link\" title=\"Enlarge\" data-key=\"71773f0cca9237f7694aadea8d9fe421\"><\/a><\/div><b>Figure 10.<\/b> The \"People\" component<\/div><\/div><\/div>People (Figure 10) need to be thoroughly educated and competent to meet the needs of the laboratory's operational procedures and scientific work. That education needs to extend beyond the traditional undergraduate program and include the specifics of instrumental techniques used. A typical four-year program doesn\u2019t have the time to cover the basic science and the practical aspects of how science is conducted in modern labs, and few schools can afford the equipment needed to meet that challenge. This broader educational emphasis is part of the intersection of people and science.\n<p>Another aspect of \u201cpeople\u201d is the development of a culture that contributes to data integrity. Lab personnel need to be educated on the organization\u2019s expectations of how lab work needs to be managed and maintained. This includes items such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_retention\" title=\"Data retention\" class=\"wiki-link\" data-key=\"d77533b92d003d39cee958a82b62391a\">records retention<\/a>, dealing with erroneous results, and what constitutes original data. They should also be fully aware of corporate and regulatory guidelines and the effort needed to enforce them.<sup id=\"rdp-ebb-cite_ref-19\" class=\"reference\"><a href=\"#cite_note-19\">[j]<\/a><\/sup> This is another instance where education beyond that provided in the undergraduate curriculum is needed.\n<\/p><p><br \/>\n<\/p><p><b>Informatics technology<\/b>\n<\/p>\n<div class=\"thumb tright\"><div class=\"thumbinner\" style=\"width:152px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig11_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"7a917f9531b90ef5bb4960b6b358b481\"><img alt=\"\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/7a\/Fig11_Liscouski_LabTechPlanMan20.png\" width=\"150\" height=\"146\" class=\"thumbimage\" \/><\/a> <div class=\"thumbcaption\"><div class=\"magnify\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig11_Liscouski_LabTechPlanMan20.png\" class=\"internal wiki-link\" title=\"Enlarge\" data-key=\"7a917f9531b90ef5bb4960b6b358b481\"><\/a><\/div><b>Figure 11.<\/b> The \"Informatics Technology\" component<\/div><\/div><\/div>Laboratory informatics technology (Figure 11) is another area where data integrity can either be enhanced or lost. The lab's digital architecture needs to be designed to support relatively easy access (within the scope of necessary security considerations) to the lab's data from the raw digitized detector output, through intermediate processed stages and to the final processed information. Unnecessary duplication of K\/D\/I must be avoided. You also need to ensure that the products chosen for lab work are suitable for the work and have the ability to be integrated electronically. After all, the goal is to avoid situations where the output of one system is printed and then manually entered into another.\n<p>The implementation and use of informatics technology should be the result of careful product selection and their intentional design\u2014from the lab bench to central database systems such as LIMS, ELN,\nSDMS, etc.\u2014rather than haphazard approach of an aggregate of lab computers.\n<\/p><p>Other areas of concern with informatics technology include backups, security, and product life cycles, which will be addressed in later sections. If as we continue onward through these goals it appears like everything touches on data integrity, it's because it does. Data integrity can be considered an optimal result of the sum of well-executed laboratory operations.\n<\/p><p><br \/>\n<\/p><p><b>The intersection points<\/b>\n<\/p><p>Two of the three intersection points deserve minor elaboration (Figure 12). First, the intersection of people and informatics technologies has several aspects the address. The first is laboratory personnel\u2019s responsibility\u2014which may be shared with corporate or LAB-IT\u2014for the selection and management of informatics products. The second is the fact that this requires those personnel to be knowledgeable concerning the application of informatics technologies in laboratory environments. Ensure the selected personnel have the appropriate backgrounds and knowledge to consider, select, and effectively use those products and technologies.\n<\/p><p>The other intersection point to be addressed is that of science with informatics technology. Here, stakeholders are concerned with product selection, system design (for automated processes), and system integration and communication with other systems and instruments. Again, as noted above, we go into more detail in later sections. The primary point here, however, can be summed up as determining whether or not the products selected for your scientific endeavors are compatible with your data integrity goals.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig12_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"9f06bfd4131804e8eb9cec219f8b813c\"><img alt=\"Fig12 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/80\/Fig12_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 12.<\/b> The intersection of People and Informatics Technology (left) and Science and Informatics Technology (right)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Addressing the needs of these two intersection points requires deliberate effort and many planning questions regarding vendor support, quality of design, system interoperability, result output, and scientific support mechanisms. Questions to ask include:\n<\/p>\n<ul><li> <b>Vendor support<\/b>: How responsive are vendors to product issues? Do you get a fast, usable response or are you left hanging? A product that is having problems can affect data quality and reliability.<\/li>\n<li> <b>Quality of design<\/b>: How easy is the system to use? Are controls, settings, and working parameters clearly defined and easily understood? Do you know what effect changes in those points will have on your results? Has the system been tuned to your needs (not adjusted to give you the answers you want, but set to give results that truly represent the analysis)? Problems with adjusting settings properly can distort results. (This is one area where data integrity may be maintained throughout a process, and then lost because of improper or untested controls on an instrument's operation.)<\/li>\n<li> <b>System interoperability<\/b>: Will there be any difficulty in integrating a software product or instrument into a workflow? Problems with sample container compatibility, operation, control software, etc. can cause errors to develop in the execution of a process flow. For example, problems with pipette tips can cause errors in fluid delivery.<\/li>\n<li> <b>Result output<\/b>: Is an electronic transfer of data possible, or does the system produce printed output (which means someone typing results into another system)? How effective is the communications protocol; is it based on a standard or does it require custom coding, which could be error prone or subject to interference? Is the format of the data file one that prevents changes to the original data? For example, CSV files allow easy editing and have the potential for corruption, nullifying data integrity efforts.<\/li>\n<li> <b>Scientific support mechanisms<\/b>: Does the product fully meet the intended need for functionality, reliability, and accuracy?<\/li><\/ul>\n<p>The underlying goal in this section goes well beyond the material that is covered in schools. Technology development in instrumentation and the application of computing and informatics is progressing rapidly, and you can\u2019t assume that everything is working as advertised, particularly for your application. Software has bugs and hardware has limitations. Applying healthy skepticism towards products and requiring proof that things work as needed protect the quality of your work. \n<\/p><p>If you\u2019re a scientist reading this material, you might wonder why you should care. The answer is simply this: it is the modern evolution of how laboratory work gets done and how results are put to use. If you don\u2019t pay attention to the points noted, data integrity may be compromised. You may also find yourself the unhappy recipient of a regulatory warning letter.\n<\/p><p>While there are some outcomes that could occur that you prefer didn't, there are also positive outcomes to come from your data integrity efforts: your work will be easier and protected from loss, results will be easier to organize and analyze, and you\u2019ll have a better functioning lab. You\u2019ll also have fewer unpleasant surprises when technology changes occur and you need to transition from one way of doing things to another. Yet there's more to protecting the integrity of your K\/I\/D than addressing the science, people, and information technology of your lab. The security of your lab and its information systems must also be addressed.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Fifth_goal:_Addressing_security_throughout_the_lab\">Fifth goal: Addressing security throughout the lab<\/span><\/h3>\n<p>Security is about protection, and there are two considerations in this matter: what are we protecting and how do we enact that protection? The first is easily answered by stating that we're protecting our ability to effectively work, as well as the results of that work. This is largely tied to the laboratory's data integrity efforts. The second consideration, however, requires a few more words.\n<\/p><p>Broadly speaking, security is not a popular subject in science, as it is viewed as not advancing scientific work or the development of K\/I\/D. Security is often viewed as inhibiting work by imposing a behavioral structure on people's freedom to do their work how they wish. Given these perceptions, it should be a lab's goal to create a functional security system that provides the protection needed while at the same time minimizing the intrusion in people\u2019s ability to work.\n<\/p><p>This section will look at a series of topics that address the physical and electronic security of laboratory work. Those major topics are shown in Figure 13 below. The depth of the commentary will vary, with some topics getting discussed at length and others by brief reference to others' work.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig13_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"02ea08d21ba9577fdbb05acb12e0a76e\"><img alt=\"Fig13 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/42\/Fig13_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 13.<\/b> The key issues of laboratory security<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Why must security be addressed in the laboratory? There are many reasons, which are best diagramed, as seen in Figure 14:\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig14_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"3e83b80cf5ca03e07b0ba68cacdd1e84\"><img alt=\"Fig14 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/41\/Fig14_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 14.<\/b> The primary reasons why security issues need to be addressed in the lab<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>All of these reasons have one thing in common: they affect our ability to work and access the results of that work. This requires a security plan. In the end, implemented security efforts either preserve those abilities, or they reduce the value and utility of the work and results, particularly if security isn't implemented well or adds a burden to personnel's ability to work. While addressing these reasons and their corresponding protections, we should keep in mind a number of issues when developing and implementing a security plan within the lab (Figure 15). Issues like remote access have taken on particular significance over the course of the COVID-19 pandemic.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig15_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"3aba7203ca3ad469373d042c947630a7\"><img alt=\"Fig15 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/12\/Fig15_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 15.<\/b> Issues to keep in mind when planning security programs<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>When the subject of security comes up, people's minds usually go in one of two directions: physical security (i.e., controlled access) and electronic security (i.e., malware, viruses, ransomware, etc.). We\u2019re going to come at it from a different angle: how do the people in your lab want to work? Instead of looking at a collection of solutions to security issues, we\u2019re going to first consider how lab personnel want to be working and within what constraints, and then we'll see what tools can be used to make that possible. Coming at security from that perspective will impact the tools you use and their selection, including everything from instrument data systems to database products, analytical tools, and cloud computing. The lab bench is where work is executed, and the planning and thinking take place between our ears, something that can happen anywhere. How do we provide people with the freedom to be creative and work effectively (something that may be different for each of us) while maintaining a needed level of physical and intellectual property security? Too often security procedures seem to be designed to frustrate work, as noted in the previous Figure 15.\n<\/p><p>The purpose of security procedures are to protect intellectual property, data integrity, resources, our ability to work, and lab personnel, all of which can be impacted by the reasons given in the prior Figure 14. However, the planning for how to approach these security procedures requires the coordination with and cooperation of several stakeholders within and tangentially related to the laboratory. Ensure these and any other necessary stakeholders are involved with the security planning efforts of your laboratory:\n<\/p>\n<ul><li> Facilities management: These stakeholders manage the physical infrastructure you are working in and have overall responsibility for access control and managing the human security assets in larger companies. In smaller companies and startups, the first line of security may be the receptionist; how well trained are they to deal with the subject?<\/li>\n<li> IT groups: These stakeholders will be responsible for designing and maintaining (along with facilities management) the electronic security systems, which range from passkeys to networks.<\/li>\n<li> Legal: These stakeholders may work with human resources to set personnel standards for security, reviewing licensing agreements and contracts\/leases for outside contractors and buildings (more later).<\/li>\n<li> Lab personnel: From the standpoint of this guide, this is all about the people doing the analytical and research work within the laboratory.<\/li>\n<li> Consultants: Security is a complex and rapidly developing subject, and you will likely need outside support to advise you on what is necessary and possible, as well as how to go about making that a reality.<\/li><\/ul>\n<p>But what else must be considered during you and your stakeholders' planning efforts? Before we can get into the specific technologies and practices that may be implemented within a facility, we need to look at the facility itself. \n<\/p>\n<h4><span class=\"mw-headline\" id=\"Examine_aspects_of_the_facility_itself\">Examine aspects of the facility itself<\/span><\/h4>\n<p>Does your company own the building you are working in? Is it leased? Is it shared with other companies in a single industrial complex? If you own the facility, life is simpler since you control everything. Working in a shared space that is leased or rented requires more planning and thought, preferably before you sign an agreement. You're likely to have additional aspects to seriously consider about your facility. Have the locks and door codes been changed since the last tenant left? Is power shared across other businesses in your building? Is the backup generator\u2014if there is one\u2014sufficient to run your systems? What fire protections are in place? How is networking managed in the facility? Are security personnel attuned to the needs of your company? Let's take a look at some of these and other questions that should be addressed.\n<\/p><p><b>Is the physical space well-defined, and does building maintenance have open access to your various spaces? <\/b>\n<\/p><p>Building codes vary from place to place. Some are very specific and strict, while others are almost live-and-let-live. One thing you want to be able to do is to define and control your organization's physical space and set up any necessary and additional protective boundaries. Physical firewalls are one way of doing that. A firewall should be a solid structure that acts as a barrier to fire propagation between your space and neighboring spaces, extending from below-ground areas to the roof. If it is a multi-level structure, the levels should be isolated. This may seems obvious, but in some single-level shared buildings (e.g., strip malls) the walls may not go to the roof to make it easier to route utilities like HVAC, power, and fire suppression. This can acts as an access point to your space.\n<\/p><p>Building maintenance is another issue. Do they have access to your space? Does that access come with formal consent or is that consent assumed as part of the lease or rental agreement? Several problems must be considered. First, know that anyone who has access to your physical space should be considered a weak point in your security. Employees should inherently have a vested interest in protecting your assets, but building maintenance is a different matter. Who vets them? Since these notes are focused on laboratory systems, who trains them about what to touch and what not to? (For example, an experiment could be ruined because maintenance personnel opened a fume hood, disturbing the airflow, despite any signage placed on the hood glass.) Consider more than just office systems in your space analysis, including other equipment that may be running after-hours that doesn\u2019t handle tampering, curiosity, or power outages well. Do you have robotics running multiple shifts or other moving equipment that might attract someone\u2019s curiosity? Security cameras would be useful, as would \u201cDo Not Enter\u201d signs. \n<\/p><p>Second, most maintenance staff will notify you (hopefully in writing) about their activities so you can plan accordingly, but what about emergency issues? If they have to fix a leak or a power problem, what are the procedures for safely shutting down systems? Do they have a contact person on your staff in case a problem occurs? Is there hazardous material on-site that requires special handling? Are the maintenance people aware of it and how to handle it? Answers to these questions should be formalized in policy and disseminated to both maintenance and security management personnel, and be made available to new personnel who may not be up to speed.\n<\/p><p><b>Is power shared across other businesses in your building?<\/b>\n<\/p><p>Shared power is another significant issue in any building environment. Unless someone paid careful attention to a lab's needs during construction, it can affect any facility. A number of issues can arise from misconfigured or unsupported power systems. Real-life examples of issues a computer support specialist friend of mine has encountered in the past include computers that:\n<\/p>\n<ul><li> were connected to the same circuit box as heavy duty garage doors. Deliveries would come in early in the morning and when the doors opened the computers crashed.<\/li>\n<li> were on the same circuit as air conditioners. The computers didn\u2019t crash, but the electrical noise and surging power use created havoc with systems operations and disk drives.<\/li>\n<li> connected to circuits that didn\u2019t have proper grounding or had separate grounding systems in the same room. Some didn\u2019t have external grounding at all. We worked on a problem with one computer-instrument system that had each device plugged into different power outlets. The computer\u2019s was grounded, but the instrument's power supply wasn\u2019t; once that was fixed everything worked.<\/li>\n<li> were too close to a radio tower. Every night when the radio system changed its antenna configuration, the computer experienced problems. Today, many devices generate radio signals that might interfere with each other. The fact that they are \u201cdigital\u201d systems doesn\u2019t matter; they are made of analog components.<\/li><\/ul>\n<p><b>Is the power clean, and is the backup generator\u2014if there is one\u2014sufficient to run your systems?<\/b>\n<\/p><p>Another problem is power continuity and quality. Laboratories depend on clean, reliable power. What will the impact of power outages\u2014lasting anywhere from seconds to days\u2014be on your ability to function? The longer end of the scale is easy; you stop working or relocate critical operations. Generators are one solution option, and we\u2019ll come back to those. The shorter outages, particularly if they are of the power up-down-up variety, are a separate issue. Networkable sensors with alarms and alerts for monitoring power, refrigeration, etc., permitting remote monitoring, may be required. Considerations for these intermittent outages include:\n<\/p>\n<ul><li> Do you know when they happened? What was their duration? How can you tell? (Again, consider sensor-based monitoring.)<\/li>\n<li> What effect did intermittent outages have on experiments that were running? Did the systems and instruments reset? Was data lost? Were in-process samples compromised?<\/li>\n<li> What effect did they have on stored samples? If samples had to be maintained under controlled climate conditions, were they compromised?<\/li>\n<li> Did power loss and power cycling cause any problems with instrumentation? How do you check?<\/li>\n<li> Did systems fail into a safe mode?<\/li><\/ul>\n<p>How real are power problems? As Ula Chrobak notes in an August 2020 Popular Science article, infrastructure failures, storms, climate change, etc. are not out of the realm of possibility; if you were in California during that time, you saw the reality first-hand.<sup id=\"rdp-ebb-cite_ref-ChrobakTheUS20_20-0\" class=\"reference\"><a href=\"#cite_note-ChrobakTheUS20-20\">[10]<\/a><\/sup>\n<\/p><p><b>If laboratory operations depend on reliable power, what steps can we take to ensure that reliability?<\/b>\nFirst, site selection naturally tops the list. You want to be somewhere that has a reputation for reliable power and rapid repairs if service is lost. A site with buried wiring would be optimal, but that only benefits you a little if the industrial park has buried wiring but is actually fed with overhead wiring. Another consideration is the age of the site: an older established site may have outdated cables that are more likely to fail. The geography is also important. Nearby rivers, lakes, or an ocean might be liable to producing floods, causing water intrusion into wiring. Also, don\u2019t overlook the potential issues associated with earthquakes or nearby industries with hazardous facilities such as chemical plants or refineries. Areas prone to severe weather conditions are an additional consideration.\n<\/p><p>Second, address the overall quality of the building and its infrastructure. This affects buildings you own as well as lease; however, the difference is in your ability to make changes. How old is the wiring? Has it been inspected? Are the grounding systems well implemented? Do you have your own electrical meters, and is your power supply isolated from other units if you are leasing? Will your computers and instruments be on circuits isolated from heavy equipment and garage doors? Make an estimate of your power requirements, then at least double it. Given that, is there sufficient amperage coming into the site to manage all your instruments, computers, HVAC systems, and freezers? How long will you be occupying that space, and is there sufficient power capacity to support potential future expansion?\n<\/p><p>Third, consider how to implement generators and battery backup power. These are obvious solutions to power loss, yet they come with their own considerations:\n<\/p>\n<ul><li> Who has control over generator implementation? If you own the building, you do. If the building is leased, the owner does, and they may not even provide generator back-up power. If not, your best bet\u2014unless you are planning on staying there for a long time\u2014is to go somewhere else; the cost of installing, permitting, and maintaining a generator on a leased site may be prohibitive. A good whole-house system can run up to $10,000, plus the cost of a fueling system. <\/li>\n<li> How much power will you need and for how long, and is sufficient fuel available? Large propane tanks may need to be buried. Diesel is another option, though fire codes may limit fuel choices in multi-use facilities. The expected duration of an outage is important, also. Often we think perhaps a few hours, but snow, ice, hurricane, tornados, and earthquakes may push that out to a week or more.<\/li>\n<li> Is the generator\u2019s output suitable for the computers and instruments in your facility? A major problem to acknowledge is electrical noise: too much and you\u2019ll create more problems than you would have if the equipment had just been shut down.<\/li>\n<li> What is the startup delay of the generator? A generator can take anywhere from a few seconds to several minutes to get up to speed and produce power. Can you afford that delay? Probably not.<\/li><\/ul>\n<p>The answer to the problems noted in the last two bullets is battery backup power. These can range from individual units that are used one-per-device, like home battery backups for computers and other equipment, to battery-walls that are being offered for larger applications. The advantage is that they can come online anywhere from instantly (i.e., always-on, online systems) to a few milliseconds for standby systems. The always-on, online options contain batteries that are constantly being charged and at the same time constantly providing power to whatever they are connected to. More expensive than standby systems, they provide clean power even from a source that might otherwise be problematic. On the other hand, standby systems are constantly charging but pass through power without conditioning; noisy power in, noisy power out until a power failure occurs.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Security_and_the_working_environment\">Security and the working environment<\/span><\/h4>\n<p>When we are looking at security as a topic, we have keep in mind that we are affecting people's ability to work. Some of the laboratory's work is done at the lab bench or on instruments (which, depending on the field you\u2019re in, could range from pH meters to telescopes). However, significant work occurs away from bench, thinking and planning wherever a thought strikes. What kind of flexibility do you want people to have? Security will often be perceived as stuff that gets in the way of personnel's ability to work, despite the fact that well-implemented security protects their work. \n<\/p><p>We need to view security as a support structure enabling flexibility in how people work, not simply as a series of barriers that frustrate people. You can begin by defining the work structure as you\u2019d like it to be, at the same time recognizing that there are two sides to lab work: the intellectual (planning, thinking, evaluating, etc.) and the performed (where you have to work with actual samples, equipment, and instruments). One can be done anywhere, while the other is performed in a specific space. The availability of computers and networks can blur the distinction. \n<\/p><p>Keeping these things in mind, any security planning should consider the following:\n<\/p>\n<ul><li> How much flexibility do personnel want in their work environment vs. what they can actually have? In some areas of work, there may be significant need for lockdown with little room for flexibility, while other areas may be pretty loose.<\/li>\n<li> Do people want to work from remote places? Does the nature of the work permit it? This can be motivated by anything from \u201cthe babysitter is sick\u201d to \u201cI just need to get away and think.\u201d<\/li>\n<li> While working remotely, do people need access to lab computers for data, files (i.e., upload\/download), or to interact with experiments? Some of this can be a matter of respecting people\u2019s time. If you have an experiment running overnight or during the weekend, it would be nice to check the status remotely instead of driving back to work.<\/li>\n<li> Do people need after-hours access to the lab facilities?<\/li><\/ul>\n<p>The answers to these planning questions lay the groundwork for hardware, software, and security system requirements. Can you support the needs of personnel, and if so, how is security implemented to make it work? Will you be using gateway systems to the lab network, with additional logins for each system, two-factor authentication, or other mechanisms? The goal is to allow people to be as productive as possible while protecting the organization's resources and meeting regulatory requirements. That said, keep in mind that unless physical and virtual access points are well controlled, others may compromise the integrity of your facility and its holdings.\n<\/p><p>Employees need to be well educated in security requirements in general and how they are implemented in your facility. They need to be a willing part of the processes and not grudgingly accepting them; that lack of willingness to work within the system is a security weak point, things people will try to circumvent. One obvious problem is with username-password combinations for computer access; rather than typing that information in, biometric features are faster and less error prone.\n<\/p><p>That said, personnel should readily accept that no system should be open to unauthorized access, and that hierarchical levels of control may be needed, depending on the type of system; some people will have access to some capabilities and not others. This type of \"role-based\" access shouldn\u2019t be viewed as a matter of trust, but rather as a matter of protection. Unless the company is tiny, senior management, for example, shouldn\u2019t have administrative system level access to database systems or robotics. If management is going to have access to those levels, ensure they know exactly what they are doing. By denying access to areas not needed in a role-based manner, you limit the ability of personnel to improperly interrogate or compromise those systems for nefarious purposes.\n<\/p><p><b>What are your security control requirements?<\/b>\n<\/p><p>Figure 16 lists some of the key areas of concern for security controls. Some we\u2019ll touch on, others we'll leave to those better informed (e.g., see Tulsi 2019<sup id=\"rdp-ebb-cite_ref-TulsiGreater19_21-0\" class=\"reference\"><a href=\"#cite_note-TulsiGreater19-21\">[11]<\/a><\/sup> and Riley 2020<sup id=\"rdp-ebb-cite_ref-RileyGitLab20_22-0\" class=\"reference\"><a href=\"#cite_note-RileyGitLab20-22\">[12]<\/a><\/sup>).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig16_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"62197432df9a7414f81fa4d22fbd9902\"><img alt=\"Fig16 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/44\/Fig16_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 16.<\/b> Key areas of concern for security controls<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><b>What is your policy on system backups?<\/b>\n<\/p><p>When it comes to your computer systems, are you backing up their K\/I\/D? If so, how often? How much K\/I\/D can you afford to lose? Look at your backups on at least three levels. First, backing up the hard drive of a computer protects against failure of that computer and drive. Backing up all of a lab\u2019s data systems to an on-site server in a separate building (or virtualized locally) protects against physical damage to the lab (e.g., fire, storms, earthquake, floods, etc.). Backing up all of a lab\u2019s data systems to a remote server (or virtualized remotely) provides even more protection against physical damage to the lab facility, particularly if the server is located someplace that won\u2019t be affected by the same problems your site may be facing. It should also be somewhere that doesn\u2019t compromise legal control over your stuff; if it is on a third-party server farm in another country, that country\u2019s laws apply to access and seizure of your files if legal action is taken against you.\n<\/p><p><b>Should your various laboratory spaces and components be internet-connected?<\/b>\n<\/p><p>When looking at your lab bench spaces, instruments, database systems, etc., determine whether they should be connected to the internet. This largely depends on what capabilities you expect to gain from internet access. Downloading updates, performing online troubleshooting with vendors, and conducting online database searches (e.g., spectra, images, etc.) are a few useful capabilities, but are they worth the potential risk of intrusion? Does your IT group have sufficient protection in place to allow access and still be protected? Note, however, any requirement for a cloud-based system would render this a moot point.\n<\/p><p>Lab systems should be protected against any intrusion, including vendors. Vendor-provided files can be downloaded to flash drives, which can then be checked for malware and integrity before being manually transferred to lab systems. Consider what is more important: convenience or data protection? This may give you more to think about when you consider your style of working (e.g., remote access). However, having trusted employees access the lab network is different than third-parties.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Summary\">Summary<\/span><\/h4>\n<p>We\u2019ve only been able to touch on a few topics; a more thorough review would require a well-maintained document, as things are changing that quickly.<sup id=\"rdp-ebb-cite_ref-23\" class=\"reference\"><a href=\"#cite_note-23\">[k]<\/a><\/sup> In many labs, security is a layered activity, where as the work of the lab is planned, security issues are then considered. We\u2019d be far better off if security planning was instead conducted in concert with lab systems planning; support for security could would then become part of the product selection criteria.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sixth_goal:_Acquiring_and_developing_.22products.22_that_support_regulatory_requirements\">Sixth goal: Acquiring and developing \"products\" that support regulatory requirements<\/span><\/h3>\n<p>Products should be supportable. That seems pretty simple, but what exactly does that mean? How do we acquire them, and more importantly, how do we develop them? The methods and procedures you develop for lab use are \u201cproducts\u201d\u2014we\u2019ll come back to that.\n<\/p><p>First, here\u2019s an analogy using an automobile. The oil pan on a car may need to be replaced if it is leaking due to damage or a failed gasket; if it isn\u2019t repaired, the oil can leak out. Some vehicles are more difficult to work on than others given their engineering. For example, replacing the oil pan in some cars requires you to lift the engine block out of the car. That same car design could also force you to move the air conditioner compressor to change spark plugs. In the end, some automobile manufactures have built a reputation for cars that are easy to service and maintain, which translates into lower repair costs and longer service life. \n<\/p><p>How does that analogy translate to the commercial products you purchase for lab use, as well as the processes and procedures you develop for your lab? The ability to effectively (i.e., with ease, a low cost, etc.) support a product has to be baked into the design from the start. It can\u2019t be retrofitted.\n<\/p><p>Let\u2019s begin with the commercial products you purchase for lab use, including instruments, computer systems, and so on. One of the purchase criteria for those items is how well they are supported: mature products should have a better support infrastructure, built up over time and use. However, that doesn\u2019t always translate to high-quality support; you may find a new product getting eager support because the vendor is heavily invested in market acceptance, working the bugs out, and using referral sites to support their sales. When it comes to supporting these commercial products, we expect to see:\n<\/p>\n<ul><li> User Guides \u2013 This should tell you how the device works, what the components are (including those you shouldn\u2019t touch), how to use the control functions, what the expected operating environment is, what you need to provide to make the item usable, and so on. For electronic devices with control signal communications and data communications, the vendor will describe how it works and how they expect it to be used, but not necessarily how to use it with third-party equipment. There are limitations of liability and implied support commitments that they prefer not to get involved with. They provide a level of capability, while it\u2019s largely left up to you to make it work in your application.<\/li>\n<li> Training materials \u2013 This will take you from opening the box, setting up whatever you\u2019ve purchased, and walking through all the features and some examples of their use. The intent is to get you oriented and familiar with using it, with the finer details located in user guides. Either this document or the user guide should tell you how to ensure that the device is installed and operating properly, and what to do if it isn\u2019t. This category can also include in-person short courses as well as online courses (an increasingly popular option as something you can do at your convenience).<\/li>\n<li> Maintenance and troubleshooting manuals \u2013 This material describes what needs to be periodically maintained (e.g, installing software upgrades, cleaning equipment, etc.) and what to do if something isn\u2019t working properly.<\/li>\n<li> Support avenues - Be it telephone, e-mail, or online chat, there are typically many different ways of reaching the vendor for help. Online support can also include a \u201cknowledgebase\u201d of articles on related topics, as well as chat functions.<\/li>\n<li> User groups \u2013 Whether conducted in-person or online, venues for giving users a chance to solve problems and present material together can also prove valuable.<\/li><\/ul>\n<p>From the commercial side of laboratory equipment and systems, support is an easy thing to deal with. If you have good products and support, people will buy them. If your support is lacking, they will go somewhere else, or you will have fostered the development of a third-party support business if your product is otherwise desirable.\n<\/p><p>From the system user\u2019s perspective, lab equipment support is a key concern. Users typically don\u2019t want to take on a support role in the lab as that isn\u2019t their job. This brings us to an interesting consideration: product life cycles. You buy something, put it to use, and eventually it has to be upgraded (particularly if it involves software) or possibly replaced (as with software, equipment, instruments, etc.). Depending on how that item was integrated into the lab\u2019s processes, this can be a painful experience or an easy one. Product life cycles are covered in more detail later in this section, but for now know they are important because they apply, asynchronously, to every software system and device in your lab. Upgrade requirements may not be driven by a change in the functionality that is important to the lab, but rather due to a change to an underlying component, e.g., the computer's operating system. The reason that this is important in a discussion about support is this: when you evaluate a vendor's support capabilities, you need to cover this facet of the work. How well do they evaluate changes in the operating system (OS) in relation to the functionality of their product? Can they advise you about which upgrades are critical and those that can be done at a more convenient time? If a change to OS or a database product occurs, how quickly do they respond?\n<\/p><p>Now that we have an idea what support means for commercial products, let\u2019s consider what support means for the \"products\"\u2014i.e., the procedures and methods\u2014developed in your lab.\n<\/p><p>The end result of a typical laboratory-developed method is a product that incorporates a process (Figure 17). This idea is nothing new in the commercial space. Fluid Management Systems, Inc. has complex sample preparation processing systems as products<sup id=\"rdp-ebb-cite_ref-FMSAutomated_24-0\" class=\"reference\"><a href=\"#cite_note-FMSAutomated-24\">[13]<\/a><\/sup>, as do instrument vendors that combine autosamplers, an instrument, and a data system (e.g., some of Agilent\u2019s PAL autosampler systems incorporate sample preparation processing as part of their design<sup id=\"rdp-ebb-cite_ref-AgilentPAL_25-0\" class=\"reference\"><a href=\"#cite_note-AgilentPAL-25\">[14]<\/a><\/sup>). Those lab methods and procedures can range from a few steps to an extensive process whose implementations can include fully manual execution steps, semi-automated steps (e.g., manual plus instrumentation), and fully automated steps. In the first two cases, execution can occur with either printed or electronic documentation, or it can be managed by a LES. However, all of these implementations are subject to regulatory requirements (commercial products are subject to <a href=\"https:\/\/www.limswiki.org\/index.php\/ISO_9000\" title=\"ISO 9000\" class=\"wiki-link\" data-key=\"53ace2d12e80a7d890ce881bc6fe244a\">ISO 9000<\/a> requirements).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig17_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"2d332579e52fb24a7257395161cd95c0\"><img alt=\"Fig17 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/59\/Fig17_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 17.<\/b> A laboratory process<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"The_importance_of_documentation\">The importance of documentation<\/span><\/h4>\n<p>Regulatory requirements and guidelines (e.g., from the FDA, EPA, ISO, etc.) have been with production and R&D for decades. However, some still occasionally question those regulations' and guidelines' application to research work. Rather than viewing them as hurdles which a lab must cross to be deemed qualified, they should be viewed as hallmarks of a well-run lab. With that perspective, they remain applicable for any laboratory.\n<\/p><p>For purposes of this guide, there is one aspect of regulatory requirements that will be emphasized here: process validation, or more specifically the end result, which is a validated process. Laboratory processes, all of which have to be validated, are essentially products for a limited set of customers; in many cases its one customer, in others the same process may be replicated in other labs as-is. The more complex the implementation, and the longer the process is expected to be in use, the more important it is to incorporate some of the tools from commercial developers into lab development (Table 1). However, regardless of development path, complete documentation is of the utmost concern. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> Process and product development documentation\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Laboratory-developed products\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Commercially developed products\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022 user requirements document<br \/>\u2022 functional specification<br \/>\u2022 system design and testing<br \/>\u2022 implementation<br \/>\u2022 installation, operational, and performance qualification<br \/>(IQ, OQ, PQ) evaluation and acceptance\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022 product requirements document<br \/>\u2022 engineering or functional specification<br \/>\u2022 product testing protocol<br \/>\u2022 product readiness for market acceptance criteria<br \/> <br \/>PLUS product support elements like:<br \/>\u2022 user guides<br \/>\u2022 training materials<br \/>\u2022 maintenance and troubleshooting guides<br \/>\u2022 support mechanisms<br \/>\u2022 user groups\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Documentation is valuable because:\n<\/p>\n<ul><li> It's educational: Quality documentation ensures those carrying out the process or maintaining it are thoroughly educated. Written documentation (with edits and <a href=\"https:\/\/www.limswiki.org\/index.php\/Audit_trail\" title=\"Audit trail\" class=\"wiki-link\" data-key=\"96a617b543c5b2f26617288ba923c0f0\">audit trails<\/a>, as appropriate) acts as a stable reference point for how things should be done. The \u201cfollow-me-and-I\u2019ll-show-you\u201d approach is flawed. That method depends on someone accurately remembering and explaining the details while having the time to actually do it, all while hoping bad habits don't creep in and become part of \u201chow it\u2019s done.\u201d<\/li>\n<li> It informs: Quality documentation that is accessible provides a reference for questions and problems as they occur. The depth of that documentation, however, should be based on the nature of the process. Even manual methods that are relatively simple need some basic elements. To be innformative, it should address numerous questions. Has the instrument calibration been accurately verified? How do you tell, and how do you correct the problem if the instrument is out of calibration? What information is provided about reagents, including their age, composition, strength, and purity? When is a technician qualified to use a reagent? How are reference materials incorporated as part of the process to ensure that it is being executed properly and consistently?<\/li><\/ul>\n<p>Note that the support documents noted in Table 1 are not usually part of process validation. The intent of process validation is to show that something works as expected once it is installed.\n<\/p><p>One aspect that hasn\u2019t been mentioned so far is how to address necessary change within processes. Any lab process is going to change over time. There may be a need for increased throughput, lower operating costs, less manual work, the ability to run over multiple shifts, etc. There may also be new technologies that improve lab operations that eventually need to get incorporated into the process. As such, planning and process documentation should describe how processes are reviewed and modified, along with any associated documentation and training. This requires the original project development to be thoroughly documented, from functionality scoping to design and implementation. By including process review and modification as part of a process allows that process to be upgraded without having to rebuild everything from scratch. This level of documentation is rarely included due to the initial cost and impact on the schedule. It will affect both, but it will also show its value once changes have to be made. In the end, by adding process review and modification mechanism, you ensure a process is supportable.\n<\/p><p>To be clear, the initial design and planning of process and methods has to be done well for a supportable product. This means keeping in mind future process review and modification even as the initial process or method is being developed. It\u2019s the difference between engineering a functional and supportable system and \u201cjust making something that works.\u201d Here are three examples:\n<\/p>\n<ul><li> One instrument system vendor, in a discussion between sessions of a meeting, described how several of his customers successfully connected a <a href=\"https:\/\/www.limswiki.org\/index.php\/Chromatography_data_system\" title=\"Chromatography data system\" class=\"wiki-link\" data-key=\"a424bb889d8507b7e8912f2faf2570c6\">chromatography data system<\/a> (CDS) to a LIMS. It was a successful endeavor until one of the systems had to be upgraded, then everything broke. The programmer had made programming changes to areas of the packages that they shouldn\u2019t have. When the upgrade occurred, those changes were overwritten. The project had to be scrapped and re-developed.<\/li>\n<li> A sample preparation robotics system was similarly implemented by creating communications between devices in ways that were less than ideal. When it came time for an upgrade to one device, the whole system failed.<\/li>\n<li> A consultant was called in to evaluate a project to interface a tensile tester to a computer, as the original developer had left the company. The consultant recommended the project be scrapped and begun anew. The original developer had not left any design documentation, the code wasn\u2019t documented, and no one knew if any of it worked or how it was supposed to work. Trying to understand someone else\u2019s programming without documentation assistance is really a matter of trying to figure out their thinking process, and that can be very difficult.<\/li><\/ul>\n<p>There are a number of reasons why problems like this exist. Examples include lack of understanding of manual and automated systems design and engineering methodologies and pressure from management (e.g., \u201chow fast can you get it done,\u201d \u201ckeep costs down,\u201d and \u201cwe\u2019ll fix it in the next version\u201d). Succumbing to these short-term views will inevitably come back to haunt you in the long-term. Upgrades, things you didn\u2019t think of when the original project was planned, and support problems all tend to highlight work that could have been done better. Another saying that frequently comes up is \u201cthere is never time to do it right, but there is always time to do it over,\u201d usually at a considerably higher cost.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Additional_considerations_in_creating_supportable_systems_and_processes\">Additional considerations in creating supportable systems and processes<\/span><\/h4>\n<p><b>Single-vendor or multi-vendor?<\/b>\n<\/p><p>Let\u2019s assume we are starting with a manual method that works and has been fully validated with all appropriate documentation. And then someone wants to change that method in order to meet a company goal such as increasing productivity or lowering operational costs. Achieving goals like those usually means introducing some sort of automation, anything from automated pipettes to instrumentation depending on the nature of the work. Even more important, if a change in the fundamental science underlying the methodology is proposed, that would also require re-validation of the process.\n<\/p><p>Just to keep things simple, let\u2019s say the manual process has an instrument in it, and you want to add an autosampler to keep the instrument fed with samples to process. This means you also need something to capture and process the output or any productivity gains on the input may be lost in handling the output. We\u2019ll avoid that discussion because our concern here is supportability. There are a couple directions you can go in choosing an add-on sampler: buy one from the same vendor as the instrument, or buy it from another vendor because it is less expensive or has some interesting features (though unless those features are critical to improving the method, they should be considered \u201cnice but not necessary\u201d).\n<\/p><p>How difficult is it going to be to make any physical and electronic (control) connections to the autosampler? Granted, particularly for devices like autosamplers, vendors strive for compatibility, but there may be special features that need attention. You need to consider not just the immediate situation but also how things might develop in the future. If you purchase the autosampler from the same vendor as the instrument and control system, they are going to ensure that things continue to work properly if upgrades occur or new generations of equipment are produced (see the product life cycle discussion in the next section). If the two devices are from different vendors, compatibility across upgrades is your issue to resolve. Both vendors will do what they can to make sure their products are operating properly and answer questions about how they function, but making them work together is still your responsibility.\n<\/p><p>From the standpoint of supportability, the simpler approach is the easiest to support. Single-vendor solutions put the support burden on them. If you use multi-vendor implementations, then all the steps in making the project work have to be thoroughly documented from the statement of need, through to the functional specifications and the finished product. The documentation may not be very long, but any assistance you can give someone who has to work with the system in the future\u2014including yourself (i.e., \u201cwhat was I thinking when I did this?\u201d)\u2014will be greatly appreciated.\n<\/p><p><b>On-device programming or supervisory control?<\/b>\n<\/p><p>Another consideration is for semi- or fully-automated systems where a component is being added or substituted. When we are looking at programmable devices, one approach is to make connections between devices via on-device programming. For example, say Device A needs to work with Device B, so programming changes are made to both to accomplish the task. While this can be made to work and be fully documented, it isn\u2019t a good choice since changing one of them (via upgrade or swap) will likely mean the programming has to be re-implemented. A better approach is to use a supervisory control system to control them both, and others that may be part of the same process. It allows for a more robust design, easier adaptations, and smoother implementation. It should be easier to support since programming changes will be limited to communications codes.\n<\/p><p><b>Third-party developers and contractors?<\/b>\n<\/p><p>Frequently, third parties are brought in to provide services that aren\u2019t available through the lab or onsite IT staff. For example, a functional specification usually describes what you want as the end result and what their product is supposed to do, not how it is supposed to do it. This is left to the developer to figure out. You need to add supportability as a requirement, that the end result not only meet regulatory requirements, but that it also is designed and documented with sufficient information to have someone unfamiliar with the project understand what would have to be done if a change were made, which also requires you to think about where changes might be made in the future. This includes considering what components might be swapped for newer technologies, handling software upgrades (and what might break as a result of them), and knowing what to do if components reach their supported end-of-life and have to be replaced.\n<\/p><p>Consulting firms may respond with \u201cif something needs to be changed or fixed, just call us, we built it,\u201d which sounds reasonable. However, suppose the people who built it aren\u2019t there anymore or aren't available because they're working on other projects. The reality is the \u201ccompany\u201d didn\u2019t build the product, people working for them did.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Product_life_cycles\">Product life cycles<\/span><\/h4>\n<p>When discussing product life cycles, whether it's digital products or hardware systems, the bottom line problem is this: what do you do when a critical product needs to be updated or replaced? This can be an easy issue or a very painful one, depending on how much thought went into the design of the original procedure using that device. It's generally easy if you had the forethought of noting \u201csomeday this is going to be replaced, so how do we simplify that?\" It's more difficult if you, through wiring or software, linked devices and systems together and then can\u2019t easily separate them, particularly if no one documented how that might be accomplished. It\u2019s all a matter of systems engineering done well.\n<\/p><p><b>Note<\/b>: This material was originally published in <i><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchgate.net\/publication\/275351757_Computerized_Systems_in_the_Modern_Laboratory_A_Practical_Guide\" target=\"_blank\">Computerized Systems in the Modern Laboratory: A Practical Guide<\/a><\/i>.\n<\/p><p>An analog pH meter\u2014as long as it has been maintained\u2014will still work today. So will double-pan balances, again as long as they have been maintained and people are proficient in their use. Old lab equipment that will still function properly has been replaced with more modern equipment due to better accuracy, ease of use, and other factors. Analog instruments can still be found operating decades after their design end-of-life. It is in the digital realm that equipment that should work (and probably does) but can\u2019t be used after a few years of service. The rate of technology change is such that tools become obsolete on the order of a half-decade. For example, rotating disks, an evolving computer staple that replaced magnetic tape drives, are now being replaced with solid-state storage.\n<\/p><p>Digital systems require two components to work: hardware (e.g., the computer, plus interfaces for the hard disk, and ports for cable connections) and software (e.g., operating systems plus software drivers to access the hardware). Both hardware packaging and operating systems are changing at an increasing rate. Hardware systems are faster, with more storage, and operating systems are becoming more complex to meet consumer demands, with a trend toward more emphasis on mobile or social computing. Those changes mean that device interfaces we rely on may not be there in the next computer you have to purchase. The RS-232 serial port, a standard for instrument connections, is being replaced with USB, Firewire, and Thunderbolt connections that give support to a much wider range of devices and simplify computer design, with more usable and less costly devices. It also means that the instrument with the RS-232 interface may not work with a new computer due to there being no RS-232 ports, and the operating system may also no longer be compatible with the instrumentation.\n<\/p><p>One aspect of technology planning in laboratory work is change management, specifically the impact of technology changes and product life cycles on your ability to work. The importance of planning around product life cycles has taken on an added dimension in the digital laboratory. Prior to the use of computers, instruments were the tools used to obtain data, which was recorded in paper laboratory notebooks. End result: getting the data and recording and managing it were separate steps in lab work. If the tools were updated or replaced, the data recorded wasn\u2019t affected. In the digital realm, changes in tools can affect your ability to work with new and old data and information. The digital-enabled laboratory requires planning, with a time horizon of decades to meet legal and regulatory requirements. The systems and other tools you use may not last for decades; in fact, they will probably change several times. However, you will have to plan for the transfer of the data and information they contain and address the issue of database access and file formats. The primary situation to avoid is having data in files that you can\u2019t read.\n<\/p><p>While we are going to begin looking at planning strategies for isolated products as a starting point, please keep in mind that in reality products do not exist in isolation. The laboratory\u2019s K\/I\/D is increasingly interconnected, and changes in one part of your overall technology plan can have implications across your lab's working technology landscape. The drive toward integration and paperless laboratories has consequences that we are not fully prepared to address. We\u2019ll start with the simpler cases and build upon that foundation.\n<\/p><p>Digital products change for a number of reasons: \n<\/p>\n<ul><li> The underlying software that support informatics applications could change (e.g., operating systems, database systems), and the layers of software that build on that base have to be updated to work properly.<\/li>\n<li> Products could see improvements due to market research, customer comments, and competitive pressures.<\/li>\n<li> Vendors could get acquired, merge with another company, or split up, resulting in products merging or one product being discarded in favor of another.<\/li>\n<li> Your company could be acquired, merge with another company, or split into two or more organizations.<\/li>\n<li> Products could simply fail.<\/li>\n<li> Your lab could require a change, replacing older technologies with systems that provide more capability.<\/li><\/ul>\n<p>In each of these cases, you have a decision to make about how K\/I\/D is going to be managed and integrated with the new system(s).\n<\/p><p>But how often do digital products change? Unfortunately, there isn't much detailed information published about changes in vendor products. Historically, operating systems used to be updated with new versions on an annual basis, with updates (e.g., bug fixes, minor changes) occurring more frequently. With a shift toward subscription services, version changes can occur more frequently. The impact of an OS version change will vary depending on the OS. Some vendors take responsibility and control for the hardware and software, and as a result, upgrades support both the hardware and OS until the vendor no longer supports new OS\nversions on older systems. Other computer systems, where the hardware and software components come from different vendors, can result in the inability to access hardware components due to an upgrade. The OS upgrade only supports certain hardware features. Support for specific add-on equipment (including components provided by the computer vendor) may require finding and reinstalling drivers from the original component vendor. As for the applications that run on operating systems, they will need to be tested with each OS version change. \n<\/p><p>Applications tend to be updated on an irregular basis, for both direct installs and for cloud-hosted solutions. Microsoft Office and Adobe\u2019s Creative Cloud products may be updated as they see a need. Since both product suites are now accessed via the internet on a subscription basis (<a href=\"https:\/\/www.limswiki.org\/index.php\/Software_as_a_service\" title=\"Software as a service\" class=\"wiki-link\" data-key=\"ae8c8a7cd5ee1a264f4f0bbd4a4caedd\">software as a service<\/a> or SaaS), user action isn\u2019t required. Lab-specific applications may be upgraded or updated as the vendor sees a need; SaaS implementations are managed by the vendor according to the vendor's internal planning. Larger, stable vendors may provide upgrades on a regular, annual basis for on-site installations. Small vendors may only update when a significant change is made, which might include new features, or when forced to because of OS changes. If those OS compatible changes aren\u2019t made, you will see yourself running software that is increasingly out-of-date. That doesn\u2019t necessarily mean it will stop working (for example, Microsoft dropped support for Windows XP in the Spring of 2014, and computers running it didn\u2019t suddenly stop). What it does mean is that if your computer hardware has to be replaced, you may not be able to re-install a working copy of the software. The working lifetime of an application, particularly a large one, can be on the order of a decade or more. Small applications depend upon market acceptance and the vendor\u2019s ability to stay in business. Your need for access to data may exceed the product's life.\n<\/p><p>The perception of the typical product life cycle runs like this: a need is perceived; product requirements are drafted; the product is developed, tested, and sold, based on market response to the product; new product requirements are determined; and the cycle continues. The reality is a bit more complicated. Figure 18 shows a more realistic view of a product\u2019s life cycle. The letters in the circles refer to key points where decisions can have an impact on your lab (\"H\" = high, \"M\" = medium, \"L\" = low).\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig18_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"2d8772c77fd3c2956352cf924ce549f3\"><img alt=\"Fig18 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/02\/Fig18_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 18.<\/b> The product life cycle<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The process begins with an initial product concept, followed by the product's development, introduction, and marketing programs, and finally its release to customers. If the product is successful, the vendor gathers customer comments, analyzes competitive technologies and any new technologies that might be relevant, and determines a need for an upgrade.\n<\/p><p>This brings us to the first decision point: is an upgrade possible with the existing product? If it is, the upgrade requirements are researched and documented and the process moves to development, with generally a low impact on users. \u201cGenerally\u201d because it depends on the nature of the product and what modifications, changes, and customizations have been made by the user. If it is an application that brings a data file in, processes it, and then saves the result in an easily accessible file format, allowing no user modifications to the application itself, \u201clow impact\u201d is a fair assessment. Statistical analysis packages, image processing, and others such applications fall into this set. Problems can arise when user modifications are overwritten by the upgrade and have to be reinstalled (only a minor issue if it is a plug-in with no programming changes) or re-implemented by making programming changes (a major problem since it requires re-validation). Normally any customization (e.g., naming database elements) and data held within an application's database should be transferred without any problems, though you do need to make some checks and evaluations to ensure that this is the case. This is the inner loop of Figure 18.\n<\/p><p>Significant problems can begin if the vendor determines that the current product generation needs to be replaced to meet market demands. If this is a hardware product (e.g., pH meter, balance, instrument, computer, etc.) there shouldn\u2019t be any immediate impact (the hardware will continue to work). However, once there is a need for equipment replacement, it becomes a different matter; we\u2019ll pick this thread up later when we discuss product retirement.\n<\/p><p>Software is a different situation. The new generation of software may not be compatible with the hardware you currently have. What you have will still work, but if there are features in the new generation that you\u2019d like to have, you may find yourself having to re-implement any software changes that you\u2019ve made to the existing system. It will be like starting over again, unless the vendor takes pains to ensure that the upgrade installation is compatible with the existing software system. This includes assurances that all the data, user programming, settings, and all the details you\u2019ve implemented to make a general product work in your environment can successfully migrate. You will also have to address user education, and plan for a transition from the old system to the new one.\n<\/p><p>One problem that often occurs with new generations of software is change in the underlying data file structures. The vendor may have determined that in order to make the next generation work and to be able to offer the features they want included, the file structure and storage formats will change. This will require you to re-map the existing file structure into the new one. You may also find that some features do not work the same as they did before and your processes have to be modified. In the past, even new versions of Microsoft Office products have had compatibility issues with older versions. In large applications such as informatics or instrument data systems (e.g., multi-user chromatography data system), changes in formats can be significant. It can have an effect on importing instrument data into informatics products. For example, some vendors and users use a formatted PDF file as a means of exchanging instrument data with a LES, SDMS, or ELN. If the new version of an instrument data system changes its report formatting, the PDF parsing routine will have be updated.\n<\/p><p>At this point, it's important to note that just because a vendor comes out with a new software or hardware package doesn\u2019t mean that you have to upgrade. If what you have is working and the new version or generation doesn\u2019t offer anything of significant value (particularly when the cost of upgrading and the impact on lab operations is factored in) then bypass the upgrade. Among the factors that can tug you into an upgrade is the potential loss of support for the products you are concerned with.\n<\/p><p>What we\u2019ve been discussing in the last few paragraphs covers the outer loop to the right of Figure 18. The next point we need to note in that figure is the \u201cNo\u201d branch from \u201cNew Product Generation Justified?\u201d and \u201cProduct Fails,\u201d both of which lead to product retirement. For both hardware and software, you face the loss of customer support and the eventual need for product replacement. In both cases, there are steps that you can take to manage the potential risks.\n<\/p><p>To begin with, unless the vendor is going out of business, they are going to want to maintain a good relationship with you. You are a current and potential future customer, and they\u2019d like to avoid bad press and problems with customer relationships. Explain how the product retirement is going to affect you and get them to work with you on managing the issue; you aren\u2019t the only one affected by this (see the commentary on user groups later). If you are successful, they will see a potential liability turn into a potential asset: you can now be a referral for the quality of their customer service and support. Realistically, however, your management of product retirement or major product changes has to occur much earlier in the process.\n<\/p><p>Your involvement begins at the time of purchase. At that point you should be asking what the vendors update and upgrade policies are, how frequently they occur, what the associated costs are, how much advanced notice they give for planning, and what level of support is provided. In addition, determine where the product you are considering lies in the product\u2019s life cycle. Ask questions such as:\n<\/p>\n<ul><li> Is it new and potentially at-risk for retirement due to a lack of market acceptance? If it is and the vendor is looking for reference sites, use that to drive a better purchase agreement. Make sure that the product is worth the risk, and be prepared in case the worst-case scenario occurs.<\/li>\n<li> Is it near the end-of-life with the potential for retirement? Look at the frequency of updates and upgrades. Are they tailing off or is the product undergoing active development?<\/li>\n<li> What is the firm\u2019s financial position? Is it running into declining sales or are customers actively seeking it? Is there talk of acquisitions or mergers, either of which can put the product's future into question?<\/li><\/ul>\n<p>You should also ask for detailed technical documents that describe where programming modifications are permitted and preserved against vendor changes, and how data will be protected, along with any tools for data migration. Once you know what the limitations are for coding changes, device additions, and so on are, the consequences of deviating from them are your responsibility; whatever you do should be done deliberately and with full awareness of their impact in the future.\n<\/p><p>One point that should be clarified during the purchase process is whether you are purchasing a product or a product license. If you a purchasing a product, you own it and can do what you like with it, at least for hardware products. Products that are combinations of hardware and software may be handled differently since the hardware won\u2019t function without the software. Licenses are \u201crights to use\u201d with benefits and restrictions. Those should be clearly understood, as well as what you can expect in terms of support, upgrades, the ability to transfer products, how products can be used, etc. If there are any questions, the time to get them answered is before you sign purchase agreements. You have the best leverage for gaining information and getting reasonable concessions that are important to you while the vendor is trying to sell you something. If you license a product, the intellectual property within the product belongs to the vendor while you own your K\/I\/D; if you decide to stop using a product, you should have the ability to extract your K\/I\/D in a usable form.\n<\/p><p>Another point: if the product is ever retired, what considerations are provided to you? For a large product, they may not be willing to offer documented copies of the code so that you can provide self-support, but a small company trying to break into the market might. It doesn\u2019t hurt to ask and get any responses in writing, don\u2019t trust someone\u2019s verbal comments; they may not be there when upgrades or product retirement occurs. Additionally, it's always beneficial to conduct negotiations on purchase and licenses in cooperation with your company's IT and legal groups. IT can advise on industry practices, and the legal department\u2019s support will be needed for any agreements.\n<\/p><p>Another direction you should take is participating in user groups. Most major vendor and products have user groups that may exist as virtual organizations on LinkedIn, Yahoo, or other forums. Additionally, they often have user group meetings at major conferences. Company-sponsored group meetings provide a means for learning about product directions, raising issues, discussing problems, etc. Normally these meeting are divided into private (registered users only) and public sessions, the former being the most interesting since they provide a means of unrestricted comments. If a new version or upgrade is being considered, it will be announced and discussed at group meetings. These will also provide a mechanism for making needs known and if a product is being retired, lobbying for support. The membership contact list will provide a resource for exchanging support dialogue, particularly if the vendor is reluctant to address points that are important to you.\n<\/p><p>If a group doesn\u2019t exist, start a virtual conference and see where it goes. If participation is active, let the vendor know about it; they may take an interest and participate or make it a corporate function. It is in a company's best interest to work with its customers rather than antagonize them. Your company\u2019s support may be needed for involvement in, or starting, user groups because of the potential for liability, intellectual property protection, or other issues. Activities performed in these types of groups can be wide-ranging, from providing support (e.g., trading advice, code, tutorials, etc.) and sharing information (e.g., where to get parts, information for out-of-warranty products) to identifying and critiquing repair options and meeting informally for conferences.\n<\/p><p>The key issue is to preserve your ability to carry out your work with as little disruption as possible. That means you have to protect your access to the K\/I\/D you\u2019ve collected, along with the ability to work with it. In this regards, software systems have one possible advantage: virtualization.\n<\/p><p><b>Virtualization: An alternative to traditional computing models<\/b>\n<\/p><p>There are situations in laboratory computing that are similar to the old joke \u201cyour teeth are in great shape but the gums have to go.\u201d The equivalent situation is running a software package and finding out that the computer hardware is failing and the software isn\u2019t compatible with new equipment. That can happen if the new computer uses a different processor than the one you are working with. An answer to the problem is a technology called virtualization. In the context of the joke, it lets you move your teeth to a new set of gums; or to put it another way, it allows you to run older software packages on new hardware and avoid losing access to older data (there are some limitations).\n<\/p><p>Briefly put, virtualization allows you to run software (including the operating system) designed for one computer on an entirely different system. An example: the Windows XP operating system and applications running on a Macintosh computer using the MAC OS X operating system via VMware\u2019s Fusion product. In addition to rescuing old software, virtualization can:\n<\/p>\n<ul><li> reduce computing costs by consolidating multiple software packages on servers;<\/li>\n<li> reduce software support issues by preventing operating system upgrades from conflicting with lab software;<\/li>\n<li> provide design options for multiple labs using informatics products without incurring hardware costs and giving up lab space to on-site computers; and<\/li>\n<li> reduce interference between software packages running on the same computer.<\/li><\/ul>\n<p>Regarding that last benefit, it's worth noting that with virtualization, adding software packages means each gets its own \u201ccomputer\u201d without additional hardware costs. Product warrantees may state the software warrantee is limited to instances where the software is installed on a \u201cclean machine\u201d (just the current operation system and that software package, nothing else). Most people put more than one application on a computer, technically voiding the warrantee. Virtualized containers let you go back to that clean machine concept without buying extra hardware.\n<\/p><p>In order to understand virtualization, we have to discuss computing, but just the basics. Figure 19 shows an arrangement of the elements. When the computer is first turned on, there are three key items engaged: the central processing unit (CPU), memory, and mass storage. The first thing that happens (after the hardware startup sequence) is that portions of the operating system are placed in the memory where the CPU can read instructions and begin working. The key point is that the operating system, applications, and files are a collection of binary data elements (words) that are passed on to the CPU.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig19_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"581b842fdbf22190f7395fe6b76cb393\"><img alt=\"Fig19 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/5f\/Fig19_Liscouski_LabTechPlanMan20.png\" width=\"362\" height=\"318\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 19.<\/b> Key elements in a computer system<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The behavior of the CPU can be emulated by a software program. We can have a program that acts like an Intel processor for example, or a processor from another vendor. If we feed that program the instructions from an application, it will execute that application. There are emulators for example, that will allow your computer to emulate an Atari 2600 game console and run Asteroids. There are also emulators for other game consoles, so your computer can behave like any game console you like, as long as you have an emulator for it. Each emulator has all the programming needed to execute copies of the original game programming. They don\u2019t wear out or break. This configuration is shown in Figure 20.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig20_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"d8a4ca1c3f8dd09c02b68769693e2ce8\"><img alt=\"Fig20 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8d\/Fig20_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 20.<\/b> Emulation on a computer<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>You have a series of game emulators, each with its collection of games. Any game emulator can be loaded into memory and execute a game from its collection; games for other emulators won\u2019t work. Each game emulator and game collection is called a container. When you want to play, you access the appropriate container and go. If the mass storage is on a shared server, other people can access the same containers and run the games on their computers without interfering with each other.\n<\/p><p>How does this apply to a computer with failing hardware that is running a data analysis program? Virtualized systems allow you to make a copy of mass storage on the computer, create a container containing the CPU emulator, OS, applications, and data files, and place it on a server for later access. The hardware no longer matters because it is being replaced with the CPU emulator. Your program\u2019s container can be copied, stored, backed up, etc. It will never wear out or grow old. When you want to run it, access the container and run it. A server can support many containers.\n<\/p><p>There are some restrictions, however. First, most computers that are purchased from stores or online come pre-loaded with an operating system. Those operating systems are OEM (original equipment manufacturer) copies of the OS whose license cost is buried in the purchase price. They can\u2019t be copied or transferred legally, and some virtual servers will recognize OEM copies and not transfer them. As a result, in order to make a virtualized container, you need a fully licensed copy of the OS. Your IT group may have corporate licenses for widely used operating systems, so that may not pose a problem. Next, recognize that some applications will require a separate license for use on a virtualized system. As frequently noted, planning ahead is key: explore this option as part of the purchase agreement: you may get a better deal. Third, it's important to note that virtualized systems cannot support real-time applications such as direct analog, clock-driven, time-critical data acquisition from an instrument. The virtualized software shares resources with other containers in a time-sharing mode, and as a result the close coordination for data acquisition will not work. Fortunately, direct data acquisition (as contrasted with computer-to-instrument communications via RS-232, USB, Ethernet, etc.) is occurring less often in favor of buffered data communications with dedicated data acquisition controllers, so this is becoming less of a problem. If you need direct computer-controlled data acquisition and experiment control, this isn\u2019t the technology for you. Finally, containerized software running on virtualized systems cannot access hardware that wasn\u2019t part of the original configuration. If the computer you are using has a piece of hardware that you\u2019d like to use but wasn\u2019t on the original virtualized computer, it won\u2019t be able to use it since it doesn\u2019t have the software drivers to access it.\n<\/p><p>If the applications software permits it, applications can have shared access to common database software. A virtualized LIMS may be a good way to implement the application since it doesn\u2019t require hardware in the lab and uses servers that are probably under IT control, and as a result the systems are backed up regularly. The major hang-up on these installations is instrument connections. IT groups tend to get very conservative about that subject. Middleware could help isolate actual instrument connections from the network and could potentially resolve the situation. The issue is part technical, part political. However, virtualized LIMS containers still prove beneficial for educational purposes. A student can work with the contents of a container, experiment as needed, and when done dismiss the container without saving it; the results of the experiments are gone, mistakes and all.\n<\/p><p>There are different types of virtualization. One has containers sharing a common emulator and operating system. As a result, you update or upgrade the emulator software and\/or operating system once and the change is made across all containers. That can cause problems for some applications; however, they can be moved to a second type of virtualization in which each container has it own copy of the operating system and they can be excluded from updates.\n<\/p><p>If you find this technology appealing, check with your vendor to see if the products of interest will function in a virtualized environment (not all will). Carefully ask questions, perhaps asking if their software will run under VMware\u2019s products or Microsoft\u2019s Desktop Virtualization products, or even Microsoft\u2019s Hyper-V server. Some vendors don\u2019t understand the difference between virtualization and client-server computing. Get any responses in writing.\n<\/p><p><b>Retirement of hardware<\/b>\n<\/p><p>Replacing retired hardware can be a challenge. If it is a stand-alone, isolated product (not connected to anything else), the problem can be resolved by determining the specifications for a replacement, conducting due diligence, etc. It is when data systems, storage, and connections to computers enter the picture that life gets interesting. For example, replacing an instrument sans data system, such as a chromatograph or spectrometer with analog and digital I\/O (sense switches not data) connections to a computer, is essentially just a hardware replacement.\n<\/p><p>Hardware interfaced to a computer has issues because of software controls and data exchanges. What appears to be the simplest and most common situation is with serial communications (RS-232, RS-xxx). Complications include:\n<\/p>\n<ul><li> Wiring: Serial communications products do not always obey conventions for wiring, so wiring changes have to be considered and tested.<\/li>\n<li> Control functions and data exchange: Interacting with serial devices via a computer requires both control functions and data exchange. There are no standards for these, so a new purchase will likely require software changes to these. That may be avoided if the replacement device (e.g., a balance) is from the same vendor as the one you currently have, and is part of a family of products. The vendor may preserve the older command set and add new commands to access new features. If that is the case, you still have a plug-compatible replacement that needs to be tested and qualified for use.<\/li>\n<li> Interfaces: Moving from an RS-232 or similar RS- device to another interface such as USB will require a new interface (although USB ports are on almost every computer) and software changes.<\/li><\/ul>\n<p>If you are using a USB device, the wiring problems go away but the command structure and data transfer issues remain. Potential software problems are best addressed when the software is first planned and designed; good design means planning ahead for change. The primary issue is control of the external device, as data formats may also change. Those points can be addressed by device-independent programming. That means placing all device-dependent commands in one place\u2014a subroutine\u2014and formatting data into a device-independent format. Doing this makes changes, testing, and other aspects easier.\n<\/p><p>Let\u2019s take a single pan balance that has two functions: <tt>tare<\/tt> and <tt>get_weight<\/tt>. Each of those has a different command sequence of characters that are sent to the balance, and each returns either a completed code or a numerical value that maybe encoded in ASCII, BCD, or binary depending on the vendor\u2019s choice. If the commands to work with the balance are scattered throughout a program, you have a lot of changes to find, make, test, and certify as working. Device-independent programming puts them in two areas: one for the <tt>tare<\/tt> command and one for the <tt>get_weight<\/tt> command, which returns a floating-point value (e.g., 1.67).\n<\/p><p>If you have to replace the device with a new one, the command codes are changed in two places, and the returned numeric code reformatted into a standard floating point value in one place. The rest of the program works with the value without any concern for its source. That allows for a lot of flexibility in choosing balances in the lab, as different units can be used for different applications with minor software adjustments.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Summary_2\">Summary<\/span><\/h4>\n<p>As noted when we first started talking about this goal, the ability to support a product has to be designed and built in, not added on. The issues can be difficult enough when you are working with one vendor. When a second or third vendor is added to the mix, you have an entirely new level of issues to deal with. This is a matter of engineering, not just science. Supportable systems and methods have to be designed, documented, engineered, and validated to be supportable. A system or method isn\u2019t supportable simply because its individual components or steps are.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Seventh_goal:_Addressing_systems_integration_and_harmonization\">Seventh goal: Addressing systems integration and harmonization<\/span><\/h3>\n<p>In the 1960s, audio stereo systems came in three forms: a packaged, integrated product that combined an AM\/FM radio tuner, turntable, and speakers; all those components but purchasable individually; and a do-it-yourself, get-out-the-soldering-iron format. The integrated products were attractive because you could just plug them into a power source and they worked. The packaging was attractive, and you didn\u2019t have to know much beyond choosing the functions you wanted to use. In terms of component quality, that was up to the manufacturer; it was rarely top-quality as the components still met the basic needs of the application at a particular price point. \n<\/p><p>Component systems appealed to a different type of customer. They wanted to pick the best components that met their budgets, trading off the characteristics of one item against another, always with the idea of upgrading elements as needed. Each components manufacturer would guarantee that their product worked, but making the entire system work was your issue. In the 1960s, everything was analog so it was a matter of connecting wires. Some went so far as to build the components from kits (lower cost), or design something and get it to work just for the fun of it. HDTV\u2019s have some of the same characteristics as component systems, as they can work out of the box, but if you want better sound or want to add a DVD or streaming box, you have to make sure that the right set of connectors exist on the product and that there is enough of them.\n<\/p><p>In the product cases noted above, there are a limited set of choices for mixing components, so user customization isn't that much of a problem. Most things work together from a hardware standpoint, but software apps are another matter. The laboratory world isn\u2019t quite so neat and tidy.\n<\/p><p>From a lab standpoint, integrated systems are attractive for a number of reasons:\n<\/p>\n<ul><li> It suggests that someone has actually thought about how the system should function and what components need to be present, and they put a working package together (may be more than one component).<\/li>\n<li> It\u2019s installed as a package: when the installation is done, all of it works, both hardware and software.<\/li>\n<li> It\u2019s been tested as a system and all the components work together.<\/li>\n<li> You have a single point of contact for support.<\/li>\n<li> If an upgrade occurs, someone (hopefully) has made sure that upgrading some portions of the system doesn\u2019t mean others are not working; an upgrade embraces the functionality of the entire system.<\/li>\n<li> The documentation addresses the entire system, from training and support to maintenance, etc.<\/li>\n<li> It should be easier to work with because the system\u2019s functionality and organization have been thought through.<\/li>\n<li> It looks nice. Someone designed a packaged system that doesn\u2019t have a number of separate boxes with wires exposed. In the lab, that may be pushing it.<\/li><\/ul>\n<p>Achieving that on a component-by-component basis may be a bit of a challenge. Obtaining an integrated system comes down to a few considerations, not the least of which is what you define as a \u201csystem.\u201d A system is a collection of elements that are used to accomplish a task. Figure 21 shows a view of overall laboratory operations.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig21_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"f033bca2810b1e921e33f13abdccb4f0\"><img alt=\"Fig21 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a2\/Fig21_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 21.<\/b> Laboratory operations and connections to other groups<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Laboratory operations have three levels of systems: the corporate level, the lab administrative level (darker grey, including office work), and the lab bench level. Our concern is going to be primarily with the latter two; however, we have to be aware that the results of lab work (both research and testing) will find their way into the corporate sphere. If the grey box is our system, where does integration fit in and what strategies are available for meeting that goal? Integration is about connecting devices and systems so that there is a smooth flow of command\/control messages, data, and information that does not depend on human intervention. We are not, for example, taking a printout from one instrument system and then manually entering it into another; that transfer should be electronic and bi-directional where appropriate. Why is this important? Improving the efficiency of lab operations, as well as ROI, has long been on the list of desired outcomes from the use of lab automation and computing. Developing integrated systems with carefully designed mechanisms for the flow, storage, and management of K\/I\/D is central to achieving those goals.\n<\/p><p>There are different strategies for building integrated systems like these. One is to create the all-encompassing computer system that does and controls everything. Think HAL in the movie 2001, or popular conceptions of an advanced AI. However, aside from the pitfalls in popular sci-fi, that isn\u2019t an advisable strategy. First, it will most likely never be finished. Trying to come up with a set of functional specifications would take years if they were ever completed. People would be constantly adding features, some conflicting, and that alone (called scope creep) would doom the process, as it has in similar situations. Even if somehow the project were completed, it would reflect the thinking of those involved at the start. In the time that the project was underway, the needs of the lab would change, and the system would be out-of-date as soon as it was turned on. If you were to develop an adaptable system, you'd again still be dealing with scope creep. Other problems would crop up too. If any component needed maintenance, the entire system could be brought to a halt and nothing would get done. Additionally, staff turn-over would be a constant source of delays as new people were brought on board and trained, and as this system would be unique, you couldn't find people with prior experience. Finally, the budget would be hard to deal with, from the initial estimate to the likely overruns.\n<\/p><p>Another approach is to redefine the overall system as a cooperative set of smaller systems, each with its own integration strategy, with the entire unit interconnected. Integrated systems in the lab world are difficult to define as a product set, since the full scope of a lab's processes is highly variable, drawing on a wide range of instruments and equipment. We can define a functional instrument system (e.g., titrators, chromatographic equipment, etc.) but sample prep variability frustrates a complete package. One place that has overcome this is the clinical chemistry market.\n<\/p><p>At this point, we have to pan back a bit and take a few additional aspects of laboratory operations into consideration. Let's look back at Figure 17 again:\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig17_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"2d332579e52fb24a7257395161cd95c0\"><img alt=\"Fig17 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/59\/Fig17_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 17.<\/b> A laboratory process<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>We are reminded that processes are what rule lab work, and the instrumentation and other equipment play an important but subservient role. We need to define the process first and then move on from there; this is the standard validation procedure. In regards to the integration of that instrumentation and equipment, any integration has to support the entire process, not just the individual instrument and equipment packages. This is one of the reasons integration is as difficult as it is. A vendor can create an instrument package, but their ability to put components together is limited to their understanding of their usage.\n<\/p><p>Unless the vendor is trying to fit in a well-defined process with enough of a market to justify the work, there is a limit to what they can do. This is why pre-validated products are available in the clinical chemistry market: the process is pre-defined and everyone does the same thing. Note that there is nothing to prevent the same thing happening in other industries; if there were a market for a product that fully implemented a particular ASTM or USP method, vendors might take notice. There is one issue with that, however. When a lab purchases laboratory instrumentation, they often buy general-purpose components. You may purchase an infrared spectrophotometer that covers a wide spectral range that can be used in a variety of applications to justify its cost. Yet you rarely, if ever, purchase one instrument for each lab process that it might be used in unless there was sufficient demand to justify it. And that's the run: if a vendor were to create an equipment package for a specific procedure, the measuring instrument would be stripped down and tailored to the application, and it may not be usable in another process. Is there enough demand for testing to warrant the development of a packaged system? If you were doing blood work, yes, because all blood testing is done the same way; it\u2019s just a question of whether or not your lab is getting enough samples. If it\u2019s ASTM xxx, maybe not.\n<\/p><p>Additionally, the development of integrated systems needs to take the real world into account. Ask whether or not the same equipment can be used for different processes. If the same equipment setup is being used in different processes with different reagents or instrument setups (e.g., different columns in chromatography), developing an integrated electro-mechanical computer system may be a viable project since all the control and electro-mechanical systems would be the same. Returning to the cookie analogy, it's the same equipment, different settings, and different dough mixes (e.g., chocolate chip, sugar cookies, etc.). You just have to demonstrate that the proper settings are being used to ensure that you are getting consistent results.\n<\/p><p>If this sounds a little confusing, it\u2019s because \u201cintegration\u201d can occur on two levels: the movement of data and information and the movement of material. On one hand, we may be talking about integration in the context of merging the sources of data and information (where they are generated) into a common area where they can be used, managed, and accessed according to the lab's and corporate's needs. Along the way, as we\u2019ve seen in the K\/I\/D discussions, different types of data and information will be produced, all of which has to be organized and coordinated. This flow is bi-directional: generated data and information in one direction, and work lists in the other. On the other hand\u2014regarding the movement of materials\u2014we may be talking about automated devices and robotics. Those two hands are joined at the measuring instrument.\n<\/p><p>We\u2019ll begin by discussing the movement of data and information: integration is a matter of bringing laboratory-generated data and information into a structured system where it can be accessed, used, and managed. That \u201csystem\u201d may be a single database or a collection of interconnected data structures. The intent in modern lab integration is that connections between the elements of that structure are electronic, not manual, and transfers may be initiated by user commands or automated processes. Yet note that someone may consider a completely manual implementation process to be \u201cintegrated,\u201d and be willing to accept slower and less efficient data transfers (that\u2019s what we had in the last century). However, that methodology doesn\u2019t give us the improvements in productivity and ROI that are desired.\n<\/p><p>The idea that integration is moving all laboratory K\/I\/D into a structured system is considerably different than the way we viewed things in the past. Previously, the goal was to accumulate lab results into a LIMS or an ELN, with the intermediate K\/I\/D (e.g., instrument data files, etc.) placed in an SDMS. That was a short-sighted attempt at considering only data storage, without fully considering the topic of data utilization. The end goal of using a LIMS or ELN to accumulate lab results was still valid\u2014particularly for further research, reporting, planning, and administrative work\u2014but it didn\u2019t deal effectively with the material those results were based on or the potential need to revisit that work.\n<\/p><p>In this discussion we\u2019ll consider the functional hub of the lab to be a LIMS and\/or ELN, with an optional SDMS; it\u2019s your lab, and you get to choose. We\u2019re referring to this as a hub for a couple of reasons. First, it is the center of K\/I\/D management, as well as planning and administrative efforts. Second, these should be the most stable information systems in the lab; durable and slow to change. Instruments and their data systems will change and be replaced as the lab\u2019s operational needs progress. As a result, the hub is where planning efforts have to begin since decisions made here have a major impact on a lab's ability to meet its goals. The choice of a cloud-based system vs. an on-site system is just one factor to consider.\n<\/p><p>Historically, laboratories have begun by putting the data- and information-generating capability in place first. That\u2019s not unusual, since new companies need data and information to drive their business development. However, what they really need is a mechanism for managing the data and information. Today, however, the place that the development of a laboratory electronic infrastructure needs to begin is with the systems that are used to collect and manage data and information. Then we can put in place the data and information generators. It\u2019s a bit like starting a production line without considering what you\u2019re going to do with all the material you\u2019re producing.\n<\/p><p>The types of data and information generators can vary greatly. Examples include:\n<\/p>\n<ul><li> a human-based reading that is recorded manually;<\/li>\n<li> a reading recorded by an instrument with limited storage and communication abilities, e.g., balances and pH meters;<\/li>\n<li> a reading recorded by a limited-functionality device, where data is recorded and stored but must be transmitted out of the machine to be analyzed; and<\/li>\n<li> a reading recorded by a combination instrument-computer, which has the ability to record, store, analyze, and output data in various forms.<\/li><\/ul>\n<p>The issue we need to deal with for each of those generators is how to plan for where the output should be stored so that it is accessible and useful. That was the problem with earlier thinking. We focused too much on where the K\/I\/D could be stored and maintained over the long term, but not enough on its ability to be used and managed. Once the analysis was done, we recognized the need to have access to the backup data to support the results, but not the ability to work with it. The next section will look at some of the ramifications of planning for those data types.\n<\/p><p><b>Planning the integration of your data generators<\/b>\n<\/p><p>There are several ramifications of planning for your data generators that need to be discussed. Before we begin, though, we need to add two additional criteria for the planning stage:\n<\/p>\n<ul><li> You should avoid duplication of K\/I\/D unless there is a clear need for it (e.g., a backup).<\/li>\n<li> In the progression from sample preparation to sample processing, to measurement, to analysis, and then to reporting, there should not be any question of both the provenance and the location of the K\/I\/D generated from that progression.<\/li><\/ul>\n<p>That said, let's look at each generator in greater detail to better understand how we plan for their integration and the harmonization of their resulting K\/I\/D.\n<\/p><p><br \/>\n1. <i>A human-based reading that is recorded manually, or recorded from an instrument with limited storage and communication abilities<\/i>\n<\/p><p><i>Examples<\/i>: The types of devices we are looking at for these generators are balances, pH meters, volt meters, single-reading <a href=\"https:\/\/www.limswiki.org\/index.php\/Spectrophotometer\" title=\"Spectrophotometer\" class=\"wiki-link\" data-key=\"6382bb48c914f3c490400c13f9eb16e6\">spectrophotometers<\/a>, etc.\n<\/p><p><i>Method of connection<\/i>: Both the manual and digital generators don\u2019t leave much choice: the integration is direct data entry into a hub system unless they are being used as part of a LES. Manual modes mean typing (with data entry verification), while digital systems provide for an electronic transfer as the method of integration.\n<\/p><p><i>Issues<\/i>: There are problems with these generators being directly tied to a hub component (see \u201cA,\u201d Figure 22). Each device or version has its own communications protocol, and the programming is specific to that model. If the item is replaced, the data transfer protocols may differ and the programming has to change. These devices are often used for single readings or weighing a sample, with the result stored in the hub, to be use in later calculations or reporting. However, even though these are single-reading instruments, things can get complicated. Problems may crop up if their measurements are part of a time series, require a weight measurement at specific time intervals, are used for measuring the weights of similar items in medical tablet uniformity testing, or used for measuring pH during a titration. Those applications wouldn\u2019t work well with a direct connection to a hub and would be better served through a separate processor (see \u201cB,\u201d Figure 22) that builds a file of measurements that could be processed, with the results sent to the hub. This creates a need to manage the file and its link to the transmitted results. An SDMS would work well, but the hub system just became more complex. Instead of a device being directly connected to a hub, we have an intermediate system connected to an SDMS and the HUB. Integration is still easily feasible, but more planning is required. Should the intermediate system take on the role of an SDMS (all the files are stored in its file structure), you would also have to provide backup and security facilities to ensure that the files weren\u2019t tampered with and secured against loss. The SDMS would be responsible for entering the results of the work into the hub. (Remember that access to the files is needed to support any questions about the results; printed versions would require re-entering the data to show that the calculations were done properly, which is time-consuming and requires verification.)\n<\/p><p><i>Diagram<\/i>:\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig22_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"b9c60537496066c9b20c0fd2b7499bb0\"><img alt=\"Fig22 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8d\/Fig22_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 22.<\/b> Representation of the movement of data and information in the lab from a human-based reading that is recorded manually, or from an instrument with limited storage and communication abilities<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><br \/>\n2. <i>A reading recorded by a limited-functionality device<\/i>\n<\/p><p><i>Examples<\/i>: Measurements are made on one or more samples, with results stored locally, which have to be transmitted to another computer for processing or viewing. Further use may be inhibited until that is done (e.g., the device may have to transmit one set of measurements before the next set can begin). Such devices include microplate readers and spectrophotometers. Some devices can be operated manually from front panel controls, or via network connections through a higher-level controller.\n<\/p><p><i>Method of connection<\/i>: Some devices may retain the old RS-232\/422 scheme for serial transmission of measurements and receiving commands, though most have transitioned to USB, Ethernet, or possibly wireless networking.\n<\/p><p><i>Issues<\/i>: Most of these devices do not produce final calculated results; that work is left to an intermediate process placed between the device and the hub system (Figure 23). As a result, integration depends on those intermediate processes controlling one or more devices, sometimes coordinating with other equipment, calculating final results, and communicating them to the hub system. Measurement files need to be kept in electronic form to make them easier to back up, copy, transmit, and generally work with. If only printed output is available, it should be scanned and an equivalent machine-readable version created and verified. Each experimental run, which may include one or more samples, should have the associated files bundled together into an archive so that all data is maintained in one place. That may be the intermediate processor\u2019s storage or an SDMS with the appropriate organization and indexing capabilities, including links back from the hub. The electronic files may be used to answer questions about how results were produced, re-run an analysis using the original or new set of algorithms, or have the results analyzed as part of a larger study.\n<\/p><p><i>Diagram<\/i>:\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig23_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"de0127f63c2520c8ed3989ee6f4b4993\"><img alt=\"Fig23 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/cd\/Fig23_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 23.<\/b> Representation of the movement of data and information in the lab from a limited-functionality device<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>3. <i>A reading recorded by a combination instrument-computer<\/i>\n<\/p><p><i>Examples<\/i>: These generators include one-to-one instrument-to-instrument data systems (IDS), many-to-one instrument-to-computer systems, <a href=\"https:\/\/www.limswiki.org\/index.php\/Nuclear_magnetic_resonance_spectroscopy\" title=\"Nuclear magnetic resonance spectroscopy\" class=\"wiki-link\" data-key=\"a05c6a4eb8775761248c099371cdb82f\">NMR spectroscopy<\/a>, chromatography, <a href=\"https:\/\/www.limswiki.org\/index.php\/Mass_spectrometry\" title=\"Mass spectrometry\" class=\"wiki-link\" data-key=\"fb548eafe2596c35d7ea741849aa83d4\">mass spectrometry<\/a>, thermal analysis, spectrophotometers, etc.\n<\/p><p><i>Method of connection<\/i>: There are several means of connection: 1) detector to computer (A\/D), 2) computer to instrument control and accessory devices such as autosamplers (via, e.g., digital I\/O, USB), and 3) computer to centralized hub systems (via, e.g., USB, Ethernet, wireless networks). Integration between the IDS is accomplished through vendor-supported <a href=\"https:\/\/www.limswiki.org\/index.php\/Application_programming_interface\" title=\"Application programming interface\" class=\"wiki-link\" data-key=\"36fc319869eba4613cb0854b421b0934\">application programming interfaces<\/a> (APIs) on both sides of the connection.\n<\/p><p><i>Issues<\/i>: The primary with these generators is managing the data structures that consist of captured detector output files, partially processed data (e.g., descriptors such as peak size, width, area, etc.) computed results, sample information, worklists, and processing algorithms. Some of this material will get transmitted to the hub, but the hub isn't generally designed to incorporate all of it. A portion of it\u2014 the content of a printed report, for example\u2014could be sent to an SDMS. However, the bulk of it has to stay with the IDS since the software needs to interpret and present the sample data contents; the data files by themselves are useless without software to unpack and make sense of them.\n<\/p><p>From a planning standpoint, you want to reduce the number of IDSs as much as possible. While chromatography is presumably the only technique that offers a choice between one-to-one and many-to-one instruments to computers, hopefully over time that list will expand to provide better data management. Consider three chromatographs, each with its own IDS. If you are looking for data, you have three systems to check, and hopefully each has its own series of unique sample IDs. Three instruments on one IDS is a lot easier to manage and search. You also have to consider backups, upgrades, general maintenance, and cost.\n<\/p><p>Moving the instrument data files to an SDMS may not be effective unless the vendor has made provision for it. The problem is data integrity. If you have the ability to move data out of the system and then re-import it, you open up the possibility of importing data that has been edited. Some vendors prohibit this sort of activity.\n<\/p><p><i>Diagram<\/i>:\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig24_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"dc19aef89e726a9be01eb1409e4d1205\"><img alt=\"Fig24 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f0\/Fig24_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 24.<\/b> Representation of the movement of data and information in the lab from a combination instrument-computer<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The above looked at generator types in isolation; however, in reality, devices and instruments are used in combinations, each producing results that have to be maintained and organized. We must look at the data sets that are generated in the course of executing an experiment or method.\n<\/p><p>The procedures associated with an experiment or method can be executed three ways: manually, using a LES, or using a robotics implementation. The real world isn\u2019t so neatly separated; manual and LES implementations may have some steps that use automated tools. The issue we need to address in planning is the creation of an \u201cexperiment data set\u201d that brings all the results produced into one package. Should questions arise about an experiment, you have a data set that can be used as a reference. That \u201cpackage\u201d may be pages in a notebook, a word processor file, or some other log. It should contain all data recorded during a procedure, or, in the case of IDS capture data, file references or pointers to that instrument data or information. You want to be able to pull up that record and be able to answer any questions that may arise about the work.\n<\/p><p>All of that may seem pretty obvious, but there is one point that needs to be addressed: the database structure, including hub systems, IDS file structures, and SDMS all have to be well defined before you accumulate a number of experiment packages. You don\u2019t want to find yourself in a situation where you have a working system of data or information storage and then have to make significant changes to it. That could mean that all previous packages have to be updated to reflect the new system, or, worse, have to deal with an \u201cold\u201d and \u201cnew\u201d system of managing experimental work.\n<\/p><p>LES systems come in two forms: stand-alone software packages, and script-based systems that are part of a LIMS or ELN. The stand-alone systems should produce the experiment record automatically with all data and pointers to IDS captured data or information. For script-based systems, the programming for the LES function has to take that into account. As for laboratory robotics, they can be viewed as an extension of a LES: instead of a person following instructions, a robot or a collection of robotic components follows its programming to carry out a process. Developing an experimental record is part of that process.\n<\/p><p>The bottom line in all of this is simple: the management architecture for your K\/I\/D has to be designed deliberately and put in place early in a lab's development. If it is allowed to be created on an as-needed basis, the resulting collection of computers and storage will be difficult to maintain, manage, and expand in an orderly fashion. At some point, someone is going to have to reorganize it, and that will be an expensive and perhaps painful process.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Harmonization\">Harmonization<\/span><\/h4>\n<p>Harmonization is a companion goal to integration. Approached with the right mindset it can reduce:\n<\/p>\n<ul><li> installation costs,<\/li>\n<li> support costs,<\/li>\n<li> education and training requirements, and<\/li>\n<li> development effort.<\/li><\/ul>\n<p>Harmonization efforts, if used inappropriately, can create strife, increasing inter-departmental friction and conflict. The general idea of harmonization is to use common hardware and software platforms to implement laboratory systems, while ensuring that move toward commonality doesn\u2019t force people to use products that are force-fits, that don\u2019t really meet the lab's needs, but serve some other agenda. The purpose of computing systems is to help people get their work done; if they need a specific product to do it, end of story. If it can be provided using common hardware and software platforms, great, but that should not be a limiting factor. If used as a guide in the development of database systems, harmonization can make it easier to access laboratory K\/I\/D across labs. It may slow down implementations because more people\u2019s opinions have to be taken into account, but the end result will be the ability to gain more use out of your K\/I\/D.\n<\/p><p>Figure 25 shows a common LIMS server supporting three different labs. Each lab has its own database structure, avoiding conflicts and unnecessary compromises in the conduct of lab work. It does benefit reduced implementation costs and support costs. While some vendors support this, others may not; see if they are willing to work a deal since there are multiple labs systems involved. If we couple this with the common structure definitions of K\/I\/D noted earlier, accessing information across labs will be more productive.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig25_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"6e7098bd863d226162005373e457ba4f\"><img alt=\"Fig25 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/23\/Fig25_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 25.<\/b> Harmonizing LIMS platforms<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>An alternative is to force everyone into one data structure, usually to reduce costs. Savings on licensing costs may be offset by development delays as multiple labs resolve conflicts in database organization, security, access control, etc. In short, keep it simple; things will work smoother and in the long run be less costly from an implementation, maintenance, and support perspective. If there is a need or desire to go through the databases for accounting purposes or other organizational requirements, the necessary material can be exported into another file structure that can be analyzed as needed. This provides a layer of security between the lab and the rest of the organization. It\u2019s basically a matter of planning how database contents are being managed with the lab and what has to be accessed from other parts of the organization.\n<\/p><p>Part of harmonization planning process involves examining how computers are paired with instruments. You may not have multiple instances of higher-priced equipment such as mass spectrometers, NMRs, or other instruments, and having a computer dedicated to each device makes sense. However there is one instrument that you may have several of: chromatographs. You can purchase a computer for each instrument, but in this case, most CDS can support multiple instruments (Figure 26).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig26_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"ae714afe1580a09ad1050e433d31f8ed\"><img alt=\"Fig26 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2b\/Fig26_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 26.<\/b> Consolidate instrument-computer connections where feasible<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>There are advantages to having multiple instruments on one computer:\n<\/p>\n<ul><li> You have only one system to support, maintain, and back up.<\/li>\n<li> All the K\/I\/D is in one system.<\/li>\n<li> The qualification or validation process is performed once, rather than having it repeated for each system.<\/li>\n<li> Overall cost is reduced.<\/li><\/ul>\n<p>This \"multiple instruments to one computer\" configuration is the result of a low data collection rate, the modest computing requirements needed to process the instrument data, and user demands on the vendors. Given the developments in computing power and distributed data acquisition and control, this many-to-one configuration should be extended to other instrument techniques, reducing costs and bringing more efficiency to the management of K\/I\/D.\n<\/p><p><b>Regarding computer systems...<\/b>\n<\/p><p>Harmonization doesn't mean that everything should run the same OS or the same version of the OS. It means doing it where possible, but not at the expense of doing lab work effectively.\n<\/p><p>With the wide diversity of products in the laboratory market, you\u2019re going to find a mix of large and small vendors. Some may be small, growing companies that are managed by a few people, and as a result, keeping up with the latest versions of operating systems and underlying software may not be critical if it doesn\u2019t affect their product's usability or performance. Their product certification on the latest version of a software platform may lag larger vendors. That means that requiring all systems to be at the same operating system level isn\u2019t realistic. Upgrading the OS may disable the software that lab personnel depend upon.\n<\/p><p><b>Regarding the data...<\/b>\n<\/p><p>During November 19-20, 2019 Pharma IQ\u2019s Laboratory Informatics Summit held a meeting on \"Data Standardization for Lab Informatics.\" The meeting highlighted the emerging <a href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\" title=\"Journal:The FAIR Guiding Principles for scientific data management and stewardship\" class=\"wiki-link\" data-key=\"e5903ddcc7734415af1d91fcd258da90\">FAIR Guiding Principles<\/a>, which state that K\/I\/D should be findable, accessible, interoperable, and reusable (FAIR). The point of mentioning this is to highlight the growing, industry-wide importance of protecting the value of the K\/I\/D that you are collecting. No matter how much it costs to produce, if you can\u2019t find the K\/I\/D you need, it has no value because it isn\u2019t usable. The same holds true if the data supporting information can\u2019t be found.\n<\/p><p>Utilization is at the core of much of what we\u2019ve been discussing. Supporting the FAIR Guiding Principles should be part of every discussion about products and what they produce, how the database is designed, and what the interoperability between labs in your organization looks like.\n<\/p><p>Another aspect of this subject is harmonizing data definitions across your organization. The same set of terms should be used to describe an object or aspect, and their database representation should be compatible, etc. The point is to make it easier to find something and make use of it.\n<\/p><p><b>Putting this all to use<\/b>\n<\/p><p>How do you apply all of this to your new lab (an easier task) or existing lab (more challenging)? This is going to be a broad-brush discussion since every lab has their own way of handling things, from its overall mission to its equipment and procedures, so you\u2019re going to have to take these points and adjust them to fit your requirements.\n<\/p><p>To start, assume you have a hub system (either a LIMS or ELN) as the center of gravity for all your K\/I\/D collection. You build your lab's K\/I\/D management infrastructure from this center of gravity outward; effectively everything revolves around the hub and radiates out from it.<sup id=\"rdp-ebb-cite_ref-26\" class=\"reference\"><a href=\"#cite_note-26\">[l]<\/a><\/sup> \n<\/p><p>For each K\/I\/D generator, ask:\n<\/p>\n<ul><li> What does it produce, which of the K\/I\/D generator types noted earlier matches, and which model is appropriate? <\/li>\n<li> Does it generate a file that has to be processed, or is it the final measurement? <\/li>\n<li> Does the device or the system supporting it have all the information needed to move it on to the next phase of the process? For example if the device is a pH meter, what is going to key the result into the next step? It will need a sample or experiment reference ID so that it knows where the result should go.<\/li><\/ul>\n<p>For each device output, ask:\n<\/p>\n<ul><li> What happens to the generated K\/I\/D and how is it used? And remember, nothing should ever get deleted.<\/li>\n<li> Is the device output a single measurement or part of a set? Will it be combined with measurements from other devices, sample IDs, and calibration information?<\/li>\n<li> Where is the best place to put it? In an intermediate server, SDMS, or hub?<\/li>\n<li> If it is a final result of that process, should it be in the hub?<\/li>\n<li> If it is an intermediate file or result, then where?<\/li>\n<li> How might it be used in the future and what is the best way to prepare for that? Files may need to be examined in audits, transferred to another group or organization<sup id=\"rdp-ebb-cite_ref-27\" class=\"reference\"><a href=\"#cite_note-27\">[m]<\/a><\/sup>, or recalculated with new algorithms. Does your system provide trace-back from final results to source data?<\/li><\/ul>\n<p>For all devices, ask: \n<\/p>\n<ul><li> Does every device that has storage and communications capability have back up procedures put in place?<\/li><\/ul>\n<p>Depending on your point of view, whether it is on the science, laboratory operations management, or lab administration, your interest in lab computing may range from \u201cnecessary evil\u201d to \u201cmakes life easier\u201d to \u201cneeded to make the lab function,\u201d or some other perspective. You may be of the opinion that all this is interesting but not your responsibility. If not yours, then who? That topic will be covered in the next write-up.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Laboratory_systems_engineers\">Laboratory systems engineers<\/span><\/h2>\n<p>If you believe that the technology planning and management considerations noted so far in this guide are important to your laboratory, it's time to ask to whom that responsibility falls upon?\n<\/p><p>The purpose of this guide has been to highlight that the practice of science has changed, become more complex, and become more dependent on technologies that demand a lot of attention. Those technologies are not only the digital systems we\u2019ve covered, but also the scientific methodologies and instrumentation whose effective use can take\u2014through increasing specialization and depth of material\u2014an entire career to learn and apply. The user of scientific computing typically views it as a tool for getting work done and not another career. Once upon a time, the scientist knowledgeable in both laboratory work and computing was necessary; if you wanted to use computers you had to understand how they worked. Today, if you tried to do that, you\u2019d find yourself spread thin across your workload, with developments happening faster in science and computers than you a single individual can keep up with.\n<\/p><p>Let's look at what a laboratory scientist would need to be able to do in order to also support their laboratory's scientific computing needs, in addition to their normal tasks (Figure 27).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig27_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"5bc3e8b033bd91aed2e64ada5b2b21a7\"><img alt=\"Fig27 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/56\/Fig27_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 27.<\/b> Partial task list for supporting laboratory scientific computing<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>On top of those tasks, the lone scientist would also have to have the following technological knowledge and personal capabilities (Figure 28):\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig28_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"67bd475f8f92193277d0127c372ce997\"><img alt=\"Fig28 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/db\/Fig28_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 28.<\/b> Partial list of technological knowledge and personal capability requirements for supporting laboratory scientific computing<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Looking at these two figures, we're realistically considering two levels of expertise: a high, overview level that can look at the broader issues and see how architectures can be constructed and applied, and specialists in areas such as robotic, etc. However, the current state of undergraduate\u2014and to a lesser extent graduate\u2014education doesn\u2019t typically have room for the depth of course work needed to cover the material noted above. Expanding your knowledge base into something that is synergistic with your current course work is straightforward; doing it with something that is from a separate discipline creates difficulties. Where do digital systems fit into your life and laboratory career?\n<\/p><p>Let's look at what the average laboratory scientist does today. Figure 29 shows the tasks that are found in modern laboratory operations in both research and testing facilities. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig29_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"810e8a66c3b1c1cf6a003f6dad9e1b4e\"><img alt=\"Fig29 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/91\/Fig29_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 29.<\/b> Typical laboratory activities in any scientific discipline<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Now let's compare that set of tasks in Figure 29 with the task emphasis provided in today's undergraduate laboratory science courses and technology courses (Figure 30 and 31).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig30_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"048fcaf47ab7284a45fca95c4a46ae44\"><img alt=\"Fig30 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/00\/Fig30_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 30.<\/b> Task-level emphasis of laboratory science in higher-education courses<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig31_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"2988e1c6bc1347513af08a16c6337ea0\"><img alt=\"Fig31 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/25\/Fig31_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 31.<\/b> Task-level emphasis of information technology in higher-education courses<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In cases where science students have access to instrumentation-computer systems, the computers are treated as \u201cblack boxes\u201d that acquire the data (data capture), process it (data processing) and report it. How those things happen is rarely if ever discussed, with no mention of analog-digital converters, sampling rates, analysis algorithms, etc. \u201cStuff happens,\u201d yet that \u201cstuff,\u201d if not properly ran with tested parameters, can turn good bench science into junk data. How would they know? Students may or may not get exposure to LIMS or ELN systems even though it would be useful for students to capture and work with their lab results, but schools may not be willing to invest in them.\n<\/p><p>IT students will be exposed to data and information management through database courses, but not at the level that LIMS and ELNs require (e.g., instrument communications and control); the rest of the tasks in Figure 31 is practically unknown to them. They\u2019d be happy to work on the computer in Figure 31, but the instrument and the instrument connections\u2014the things that justify the computer's role\u2014aren\u2019t something they\u2019d be exposed to.\n<\/p><p>What we need are people with a foot in both fields, able to understand and be conversant in both the laboratory science and IT worlds, relating them to each other to the benefit of lab operation effectiveness while guiding IT in performing their roles. We need \u201claboratory systems engineers\u201d (LSEs).\n<\/p><p>Previously referred to as \"laboratory automation engineers\" (LAEs)<sup id=\"rdp-ebb-cite_ref-LiscouskiAreYou06_10-1\" class=\"reference\"><a href=\"#cite_note-LiscouskiAreYou06-10\">[6]<\/a><\/sup> and \"LAB-IT\" specialists, we now realize both titles fall short of the mark. \"Laboratory automation engineer\" emphasizes automation too strongly when the work is much broader than that. And \"LAB-IT\" is a way of nudging IT personnel into lab-related work without really addressing the full scope of systems that exist in labs, including robotics and data acquisition and control.\n<\/p><p>Laboratory information technology support differs considerably from classical IT work (Figure 32). The differences are primarily two-fold. First, the technologies used in lab work, including those in which instruments are attached to computers and robotics, are different than those commonly encountered in classical IT. The computers are the same, but the added interface and communications requirements imposed by instrument-computer connections change the nature of the work. When troubleshooting, it can be difficult to separate computer issues from those resulting from the connection to instruments and digital control systems. Second, the typical IT specialist, maybe straight out of school, doesn\u2019t have a frame of reference for understanding what they are dealing with in a laboratory setting. The work is foreign, the discussions involve terminology they may not understand, and there may be no common ground for discussing problems. In classical IT, the IT personnel may be using the same office software as the people they support, but they can't say the same for the laboratory software used by scientists.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig32_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"9008319c4b611d20d323042107195233\"><img alt=\"Fig32 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/25\/Fig32_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 32.<\/b> Comparison of corporate IT with laboratory IT (LAB-IT)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Having noted the differences between classing IT and laboratory IT, as well as the growing need for competent LSEs, we need to take a closer look at some of the roles that classic IT and LSE personnel can take. Figure 33 provides a sub-set of the items from Figure 32 and reflects tasks that IT groups could be comfortable with. Typical IT backgrounds with no lab tech familiarity won\u2019t get you beyond the basic level of support. To be effective, IT personnel need to become familiar with the lab environment, the applications and technologies used, and the language of laboratory work. It isn\u2019t necessary for IT support to become experts in instrumental techniques, but they should understand the basic \"instrument to control system to computer\" model as well as the related database applications, to the point where they can provide support, advise people on product selections, etc. We need people who can straddle the IT-laboratory application environment. They could be lab people with an interest in computing or IT people with a strong interest in science.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig33_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"689c853ced4614a2f9944f38b1349b14\"><img alt=\"Fig33 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/fa\/Fig33_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 33.<\/b> Potential roles for IT and LSE support in laboratory work<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>There are ways of bridging that education gap (Figure 34), but today they depend upon individual initiative more than corporate direction to educate people to the level needed. On-the-job training is not an effective substitute for real education; on the surface it is cheaper, but you lose out in the long run because people really don\u2019t understand what is going on, which limits their effectiveness and prevents them from being innovative or even catching problems in the early stages before they become serious. A big issue is this: due to a lack of education, are people developing bad K\/I\/D and aren't aware of it? The problem isn\u2019t limited to the level of systems we are talking about here. It also extends to techniques such as pipetting.<sup id=\"rdp-ebb-cite_ref-BradshawTheImpo12_28-0\" class=\"reference\"><a href=\"#cite_note-BradshawTheImpo12-28\">[15]<\/a><\/sup>\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig34_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"e1bd516cc4e1a716178147a5712de6bf\"><img alt=\"Fig34 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2e\/Fig34_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 34.<\/b> Bridging the education gap<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>It is also a matter of getting people to understand the breadth of material they have to be familiar with. In 2018, a webinar series was created (Figure 35) to educate management on the planning requirements for implementing lab systems. The live sessions were well attended. The chart shows the viewing rate for the individual topics through early December 2020. Note that the highest viewed items were technology-specific; people wanted to know about LIMS, ELN, etc. The details about planning, education, support, etc. haven\u2019t received near the amount of attention they need. People want to know about product classes but aren\u2019t willing to learn about what it takes to be successful. Even if you are relying on vendors or consultants, lab management is still accountable for the success of planning, implementation, and effectiveness of lab systems.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig35_Liscouski_LabTechPlanMan20.png\" class=\"image wiki-link\" data-key=\"1d5fe54d86f59d9502cbbfd01a557035\"><img alt=\"Fig35 Liscouski LabTechPlanMan20.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f5\/Fig35_Liscouski_LabTechPlanMan20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 35.<\/b> Laboratory technology webinar series views after initial release<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Prior to the COVID-19 pandemic of 2020, undergraduate education depended on the standard model of in-person instruction. With the challenges of COVID-19 spreading, online learning took on a new importance and stronger acceptance, building on the ground established by online universities and university programs. This gives us an acceptable model for two types of course development: a fully-dedicated LSE program or an expanded program that would expand student\u2019s backgrounds in both the laboratory sciences and IT. One issue that would need to be addressed, however, is bridging the gap between presentation material and hands-on experience with lab systems. Videos and evaluation tests will only get you so far; you need the hands-on experience to make it real and provide the confidence that what you've learned can be effectively applied.\n<\/p><p>There are several steps that can be taken to build an LSE program. The first is to develop a definition of a common set of skills and knowledge that an LSE should have, recognizing that people will come from two different backgrounds (i.e., laboratory science and IT), and those have to be built up to reach a common balanced knowledge base. Those with a strong laboratory science background need to add information technology experience, while those from IT will need to gain an understanding of how laboratory science is done. Remember, however, that those from IT experiences don\u2019t need to be educated in chemistry, biology, and physics, etc. After all, they aren\u2019t going to be developing methods; they will be helping to implement them. There are things common to all sciences that they need to understand such as record keeping, the workflow models of testing and research, data acquisition and processes, instrumentation, and so on. That curriculum should also help people who want to specialize in particular subject areas such as laboratory database systems, robotics, etc. The second step is to build a curriculum that allows students to meet those requirements. This requires solid forethought in the development and curation of course materials. A lot of material already exists and is spread over the internet on university, government, and company web sites. A good first step would be to collect and organize those references into a single site (the actual courses need not be moved to the site, just their descriptions, access requirements, and links). Presentation and organization of the content is also important. Someone visiting the site will need a guide of what LSE is about, how to find material appropriate for different subject areas, and how to get access to it. Consider your site audience a visitor that knows nothing about the field: where do they start and how do we facilitate their progress? Providing clear orientation and direction are key. First give them an understanding of what LSE is all about, and then a map to whatever interests them. With the curriculum built, you can then identify areas that need more material and then move to further develop the program. Of course, you'll also want to make it possible to take advantage of online demonstration systems and simulators to give people a feel for working with the various laboratory systems. This is a half-step to what is needed: there\u2019s no substitute for hands-on work with equipment.\n<\/p><p>As it stands today, we\u2019ve seemingly progressed from manual methods, to computer-assisted methods, and then to automated systems in the course of developing laboratory technologies over the years, and yet our educational programs are a patchwork of courses largely driven by individual needs. We need to take a new look at lab technologies and their use and how best to prepare people for their work with solid educational opportunities.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Closing\">Closing<\/span><\/h2>\n<p>This guide has addressed the following:\n<\/p>\n<ul><li> <i>Why technology planning and management needs to be addressed<\/i>: Because integrated system need attention in their application and management to protect electronic laboratory K\/I\/D, ensure that it can be effectively used, and ensure that the systems and products put in place are both the right ones, and that they fully contribute to improvements in lab operations.<\/li><\/ul>\n<ul><li> <i>What's changed about that planning and management since the introduction of computers in the lab<\/i>: As technology in the lab expanded, we lost the basic understanding of what the new computer and instrument system were and what they did, that they had faults, and that if we didn\u2019t plan for their effective use and counter those faults, we were opening ourselves to unpleasant surprises. The consequences at times were system crashes, lost data, and a lack of a real understanding of how the output of an instrument was transformed into a set of numbers, which meant we couldn\u2019t completely account for the results we were reporting. A more purposeful set of planning and management activities, at the earliest point possible, have become increasingly more important.<\/li><\/ul>\n<ul><li> <i>Why developing an environment that fosters productivity and innovation is important<\/i>: Innovation doesn\u2019t happen in a highly structured environment: you need the freedom to question, challenge, etc. You also need the tools to work with. The inspiration that leads to innovation can happen anywhere, anytime. All of a sudden all the pieces fit. This requires flexibility and trust in people, an important part of corporate culture.<\/li><\/ul>\n<ul><li> <i>Why developing high-quality K\/I\/D is desirable<\/i>: There are different types of data structures that are used in lab work, and careful attention is needed to work with and manage them. This includes the effective management of K\/I\/D, putting it in a structure that encourages its use and protects its value. When methods are proven and you have documented evidence that they were executed by properly educated personnel using qualified reagents, instruments, and methods, you should then have high-quality K\/I\/D to support each sample result and any other information gleaned from that data.<\/li><\/ul>\n<ul><li> <i>Why fostering a culture around data integrity is important to lab operations, addressing both technical and personnel issues<\/i>: Positive outcomes will come from your data integrity efforts: your work will be easier and protected from loss, results will be easier to organize and analyze, and you\u2019ll have a better functioning lab. You\u2019ll also have fewer unpleasant surprises when technology changes occur and you need to transition from one way of doing things to another.<\/li><\/ul>\n<ul><li> <i>How to address digital, facility, and backup security<\/i>: Preventing unauthorized electronic and physical intrusion is critical to data integrity and meeting regulatory requirements. It also ensures that access to K\/I\/D is protected against loss from a wide variety of threats to the organization's facilities, all while securing your ability to work. This included addressing power backup, continuity of operations, systems backup, and more.<\/li><\/ul>\n<ul><li> <i>How to acquire and develop \"products\" that support regulatory requirements<\/i>: Careful engineering and well-planned and -documented internal processes are needed to ensure that systems and methods that are being used can remain in use and be supported over the life span of a lab. This means recognizing the initial design and planning of processes and methods has to be done well for a supportable product, and keeping in mind the potential for future process review and modification even as the initial process or method is being developed. Additionally, the lab must also recognize the complete product life cycle and how that affects the supportability of systems and methods.<\/li><\/ul>\n<ul><li> <i>The importance of system integrations and the harmonization of K\/I\/D<\/i>: Integrated systems can benefit a lab's operations and the planning needed to work with different types of database systems as the results of lab work becoming more concentrated in LIMS and ELNs, including making decisions about how K\/I\/D is stored and distributed over multiple databases. At the same time, harmonization efforts using common hardware and software platforms to implement laboratory systems is important, but those efforts must also ensure that the move toward commonality doesn\u2019t force people to use products that are forced fits, that don\u2019t really meet the lab's needs, but serve some other agenda. <\/li><\/ul>\n<ul><li> <i>Why the development of comprehensive higher-education courses dedicated to the laboratory systems engineer or lab science-IT hybrid is a must with today's modern laboratory technology<\/i>: In today's world, typical IT backgrounds with no lab tech familiarity, or typical laboratory science backgrounds with no IT familiarity won\u2019t get you beyond the basic level of support for your laboratory systems. To be effective, IT personnel need to become familiar with the lab environment, the applications and technologies used, and the language of laboratory work, while scientists must become more familiar with the management of K\/I\/D from the technical perspective. This gap must be closed through new and improved higher-education programs.<\/li><\/ul>\n<p>And thus we return back to the start to close this guide. First, there's a definite need for better planning and management of laboratory technologies. Careful attention is required in to protect electronic laboratory knowledge, information, and data (K\/I\/D), ensure that it can be effectively used, and ensure that the systems and products put in place are both the right ones, and that they fully contribute to improvements in lab operations. Second, seven clear goals highlight this apparent need for laboratory technology planning and management and improve how it's performed. From supporting an environment that fosters productivity and innovation all the way to ensuring proper systems integration and harmonization, planning and management is a multi-step process with many clear benefits. And finally, there's a definitive need for more laboratory systems engineers (LSEs) who have the education and skills needed to accomplish all that planning and management in an effective manner, from the very start. This will require a more concerted effort in academia, and perhaps even among professional organizations catering to laboratories. All of this together hopefully means a more thoughtful, modern, and deliberate approach to implementing laboratory technologies in your lab.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations.2C_acronyms.2C_and_initialisms\">Abbreviations, acronyms, and initialisms<\/span><\/h2>\n<p><b>A\/D<\/b>: Analog-to-digital\n<\/p><p><b>AI<\/b>: Artificial intelligence\n<\/p><p><b>ALCOA<\/b>: Attributable, legible, contemporaneous, original, and accurate\n<\/p><p><b>API<\/b>: Application programming interface\n<\/p><p><b>CDS<\/b>: Chromatography data system\n<\/p><p><b>CPU<\/b>: Central processing unit\n<\/p><p><b>ELN<\/b>: Electronic laboratory notebook\n<\/p><p><b>EPA<\/b>: Environmental Protection Agency\n<\/p><p><b>FAIR<\/b>: Findable, accessible, interoperable, and reusable\n<\/p><p><b>FDA<\/b>: Food and Drug Administration\n<\/p><p><b>FRB<\/b>: Fast radio bursts\n<\/p><p><b>IT<\/b>: Information technology\n<\/p><p><b>ISO<\/b>: International Organization for Standardization\n<\/p><p><b>K\/D\/I<\/b>: Knowledge, data, and information\n<\/p><p><b>LAB-IT<\/b>: Laboratory information technology support staff\n<\/p><p><b>LAE<\/b>: Laboratory automation engineering (or engineer)\n<\/p><p><b>LES<\/b>: Laboratory execution system\n<\/p><p><b>LIMS<\/b>: Laboratory information management system\n<\/p><p><b>LIS<\/b>: Laboratory information system\n<\/p><p><b>LOF<\/b>: Laboratory of the future\n<\/p><p><b>LSE<\/b>: Laboratory systems engineer\n<\/p><p><b>ML<\/b>: Machine learning\n<\/p><p><b>OS<\/b>: Operating system\n<\/p><p><b>QA\/QC<\/b>: Quality assurance\/quality control\n<\/p><p><b>ROI<\/b>: Return on investment\n<\/p><p><b>SDMS<\/b>: Scientific data management system\n<\/p><p><b>SOP<\/b>: Standard operating procedure\n<\/p><p><b>TPM<\/b>: Technology planning and management\n<\/p><p><br \/>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Footnotes\">Footnotes<\/span><\/h2>\n<div class=\"reflist\" style=\"list-style-type: lower-alpha;\">\n<ol class=\"references\">\n<li id=\"cite_note-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-1\">\u2191<\/a><\/span> <span class=\"reference-text\">See <i><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13140\/RG.2.2.15605.42724\" target=\"_blank\">Elements of Laboratory Technology Management<\/a><\/i> and the LSE material in this document.<\/span>\n<\/li>\n<li id=\"cite_note-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-4\">\u2191<\/a><\/span> <span class=\"reference-text\">See the \"Scientific Manufacturing\" section of <i><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13140\/RG.2.2.15605.42724\" target=\"_blank\">Elements of Laboratory Technology Management<\/a><\/i>.<\/span>\n<\/li>\n<li id=\"cite_note-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-8\">\u2191<\/a><\/span> <span class=\"reference-text\">By \u201cgeneral systems\u201d I\u2019m not referring to simply computer systems, but the models and systems found under \u201cgeneral systems theory\u201d in mathematics.<\/span>\n<\/li>\n<li id=\"cite_note-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-9\">\u2191<\/a><\/span> <span class=\"reference-text\">Regarding LAB-IT and LAEs, my thinking about these titles has changed over time; the last section of this document \u201cLaboratory systems engineers\u201d goes into more detail.<\/span>\n<\/li>\n<li id=\"cite_note-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-11\">\u2191<\/a><\/span> <span class=\"reference-text\">\u201cReal-time\u201d has a different meaning inside the laboratory than it does in office applications. Instead of a response of a couple seconds between an action and response, lab \u201creal-time\u201d is often a millisecond or faster precision; missing a single sampling timing out of thousands can invalidate an entire sample analysis.<\/span>\n<\/li>\n<li id=\"cite_note-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-12\">\u2191<\/a><\/span> <span class=\"reference-text\">See <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:Notes_on_Instrument_Data_Systems\" title=\"LII:Notes on Instrument Data Systems\" class=\"wiki-link\" data-key=\"1b7330228fd59158aab6fab82ad0e7cc\">Notes on Instrument Data Systems<\/a><\/i> for more on this topic.<\/span>\n<\/li>\n<li id=\"cite_note-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-14\">\u2191<\/a><\/span> <span class=\"reference-text\">For a more detailed description of the K\/I\/D model, please refer to <i><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchgate.net\/publication\/275351757_Computerized_Systems_in_the_Modern_Laboratory_A_Practical_Guide\" target=\"_blank\">Computerized Systems in the Modern Laboratory: A Practical Guide<\/a><\/i>.<\/span>\n<\/li>\n<li id=\"cite_note-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-15\">\u2191<\/a><\/span> <span class=\"reference-text\">For more detailed discussion on this, see <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:Notes_on_Instrument_Data_Systems\" title=\"LII:Notes on Instrument Data Systems\" class=\"wiki-link\" data-key=\"1b7330228fd59158aab6fab82ad0e7cc\">Notes on Instrument Data Systems<\/a><\/i>.<\/span>\n<\/li>\n<li id=\"cite_note-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-16\">\u2191<\/a><\/span> <span class=\"reference-text\">For more information on virtualization, particularly if the subject is new to you, look at <i><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.vmware.com\/content\/dam\/learn\/en\/amer\/fy20\/pdf\/50872_20Q1_Next-Gen_Virtualization_FD_VMware_Special_Edition.pdf\" target=\"_blank\">Next-Gen Virtualization for Dummies<\/a><\/i>. The <i>for Dummies<\/i> series is designed to educate people new to a topic, getting away from jargon and presenting material in clear, easy-to-understand language. This book is particularly good at that.<\/span>\n<\/li>\n<li id=\"cite_note-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-19\">\u2191<\/a><\/span> <span class=\"reference-text\">One good reference on this subject is a presentation <i><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.pharma-iq.com\/events-laboratory-informatics-online\/downloads\/building-a-data-integrity-strategy-to-accompany-your-digital-enablement\" target=\"_blank\">Building A Data Integrity Strategy To Accompany Your Digital Enablement<\/a><\/i> by Julie Spirk Russom of BioTherapeutics Pharmaceutical Science.<\/span>\n<\/li>\n<li id=\"cite_note-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-23\">\u2191<\/a><\/span> <span class=\"reference-text\">Though it may not see significant updates, consider reading the <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:Comprehensive_Guide_to_Developing_and_Implementing_a_Cybersecurity_Plan\" title=\"LII:Comprehensive Guide to Developing and Implementing a Cybersecurity Plan\" class=\"wiki-link\" data-key=\"fce1737a2e9697fc03e956327817f8ea\">Comprehensive Guide to Developing and Implementing a Cybersecurity Plan<\/a><\/i> for a much more comprehensive look at security in the lab.<\/span>\n<\/li>\n<li id=\"cite_note-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-26\">\u2191<\/a><\/span> <span class=\"reference-text\">Yes the scientific work you do is essential to the lab\u2019s purpose, but our focus is on one element of the lab\u2019s operations: what happens after the scientific work is done.<\/span>\n<\/li>\n<li id=\"cite_note-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-27\">\u2191<\/a><\/span> <span class=\"reference-text\">For example, a product line is sold to another company or transferred to another division, and they then want copies of all relevant information. Meeting regulatory requirements is another example.<\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"About_the_author\">About the author<\/span><\/h2>\n<p>Initially educated as a chemist, author Joe Liscouski (joe dot liscouski at gmail dot com) is an experienced laboratory automation\/computing professional with over forty years of experience in the field, including the design and development of automation systems (both custom and commercial systems), LIMS, robotics and data interchange standards. He also consults on the use of computing in laboratory work. He has held symposia on validation and presented technical material and short courses on laboratory automation and computing in the U.S., Europe, and Japan. He has worked\/consulted in pharmaceutical, biotech, polymer, medical, and government laboratories. His current work centers on working with companies to establish planning programs for lab systems, developing effective support groups, and helping people with the application of automation and information technologies in research and quality control environments.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-BourneMyBoss13-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BourneMyBoss13_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bourne, D. (2013). \"My boss the robot\". <i>Scientific American<\/i> <b>308<\/b> (5): 38\u201341. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fscientificamerican0513-38\" target=\"_blank\">10.1038\/scientificamerican0513-38<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23627215\" target=\"_blank\">23627215<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=My+boss+the+robot&rft.jtitle=Scientific+American&rft.aulast=Bourne%2C+D.&rft.au=Bourne%2C+D.&rft.date=2013&rft.volume=308&rft.issue=5&rft.pages=38%E2%80%9341&rft_id=info:doi\/10.1038%2Fscientificamerican0513-38&rft_id=info:pmid\/23627215&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CookCollab20-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CookCollab20_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cook, B. (2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labmanager.com\/laboratory-technology\/collaborative-robots-mobile-and-adaptable-labmates-24474\" target=\"_blank\">\"Collaborative Robots: Mobile and Adaptable Labmates\"<\/a>. <i>Lab Manager<\/i> <b>15<\/b> (11): 10\u201313<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.labmanager.com\/laboratory-technology\/collaborative-robots-mobile-and-adaptable-labmates-24474\" target=\"_blank\">https:\/\/www.labmanager.com\/laboratory-technology\/collaborative-robots-mobile-and-adaptable-labmates-24474<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Collaborative+Robots%3A+Mobile+and+Adaptable+Labmates&rft.jtitle=Lab+Manager&rft.aulast=Cook%2C+B.&rft.au=Cook%2C+B.&rft.date=2020&rft.volume=15&rft.issue=11&rft.pages=10%E2%80%9313&rft_id=https%3A%2F%2Fwww.labmanager.com%2Flaboratory-technology%2Fcollaborative-robots-mobile-and-adaptable-labmates-24474&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HsuIsIt18-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HsuIsIt18_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Hsu, J. (24 September 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nbcnews.com\/mach\/science\/it-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586\" target=\"_blank\">\"Is it aliens? Scientists detect more mysterious radio signals from distant galaxy\"<\/a>. <i>NBC News MACH<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nbcnews.com\/mach\/science\/it-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586\" target=\"_blank\">https:\/\/www.nbcnews.com\/mach\/science\/it-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Is+it+aliens%3F+Scientists+detect+more+mysterious+radio+signals+from+distant+galaxy&rft.atitle=NBC+News+MACH&rft.aulast=Hsu%2C+J.&rft.au=Hsu%2C+J.&rft.date=24+September+2018&rft_id=https%3A%2F%2Fwww.nbcnews.com%2Fmach%2Fscience%2Fit-aliens-scientists-detect-more-mysterious-radio-signals-distant-galaxy-ncna912586&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TimmerAIPlus18-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TimmerAIPlus18_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Timmer, J. (18 July 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/arstechnica.com\/science\/2018\/07\/ai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work\/5\/\" target=\"_blank\">\"AI plus a chemistry robot finds all the reactions that will work\"<\/a>. <i>Ars Technica<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/arstechnica.com\/science\/2018\/07\/ai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work\/5\/\" target=\"_blank\">https:\/\/arstechnica.com\/science\/2018\/07\/ai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work\/5\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=AI+plus+a+chemistry+robot+finds+all+the+reactions+that+will+work&rft.atitle=Ars+Technica&rft.aulast=Timmer%2C+J.&rft.au=Timmer%2C+J.&rft.date=18+July+2018&rft_id=https%3A%2F%2Farstechnica.com%2Fscience%2F2018%2F07%2Fai-plus-a-chemistry-robot-finds-all-the-reactions-that-will-work%2F5%2F&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HelixAIHome-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HelixAIHome_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.askhelix.io\/\" target=\"_blank\">\"HelixAI - Voice Powered Digital Laboratory Assistants for Scientific Laboratories\"<\/a>. HelixAI<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.askhelix.io\/\" target=\"_blank\">http:\/\/www.askhelix.io\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=HelixAI+-+Voice+Powered+Digital+Laboratory+Assistants+for+Scientific+Laboratories&rft.atitle=&rft.pub=HelixAI&rft_id=http%3A%2F%2Fwww.askhelix.io%2F&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiscouskiAreYou06-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiscouskiAreYou06_10-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiscouskiAreYou06_10-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liscouski, J.G. (2006). \"Are You a Laboratory Automation Engineer?\". <i>SLAS Technology<\/i> <b>11<\/b> (3): 157-162. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jala.2006.04.002\" target=\"_blank\">10.1016\/j.jala.2006.04.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Are+You+a+Laboratory+Automation+Engineer%3F&rft.jtitle=SLAS+Technology&rft.aulast=Liscouski%2C+J.G.&rft.au=Liscouski%2C+J.G.&rft.date=2006&rft.volume=11&rft.issue=3&rft.pages=157-162&rft_id=info:doi\/10.1016%2Fj.jala.2006.04.002&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiscouskiWhich15-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiscouskiWhich15_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liscouski, J. (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchgate.net\/publication\/291971749_Which_Laboratory_Software_is_the_Right_One_for_Your_Lab\" target=\"_blank\">\"Which Laboratory Software Is the Right One for Your Lab?\"<\/a>. <i>PDA Letter<\/i> (November\/December 2015): 38\u201341<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.researchgate.net\/publication\/291971749_Which_Laboratory_Software_is_the_Right_One_for_Your_Lab\" target=\"_blank\">https:\/\/www.researchgate.net\/publication\/291971749_Which_Laboratory_Software_is_the_Right_One_for_Your_Lab<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Which+Laboratory+Software+Is+the+Right+One+for+Your+Lab%3F&rft.jtitle=PDA+Letter&rft.aulast=Liscouski%2C+J.&rft.au=Liscouski%2C+J.&rft.date=2015&rft.issue=November%2FDecember+2015&rft.pages=38%E2%80%9341&rft_id=https%3A%2F%2Fwww.researchgate.net%2Fpublication%2F291971749_Which_Laboratory_Software_is_the_Right_One_for_Your_Lab&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WPDataInt-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WPDataInt_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"nofollow\" class=\"external text wiki-link\" href=\"https:\/\/en.wikipedia.org\/wiki\/Data_integrity\" data-key=\"42abdc13351f50028f23f899197d3349\">\"Data integrity\"<\/a>. <i>Wikipedia<\/i>. 03 February 2021<span class=\"printonly\">. <a rel=\"nofollow\" class=\"external free wiki-link\" href=\"https:\/\/en.wikipedia.org\/wiki\/Data_integrity\" data-key=\"42abdc13351f50028f23f899197d3349\">https:\/\/en.wikipedia.org\/wiki\/Data_integrity<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+integrity&rft.atitle=Wikipedia&rft.date=03+February+2021&rft_id=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FData_integrity&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarmonWhatIs20-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HarmonWhatIs20_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Harmon, C. (20 November 2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.technologynetworks.com\/informatics\/articles\/what-is-data-integrity-343068\" target=\"_blank\">\"What Is Data Integrity?\"<\/a>. <i>Technology Networks<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.technologynetworks.com\/informatics\/articles\/what-is-data-integrity-343068\" target=\"_blank\">https:\/\/www.technologynetworks.com\/informatics\/articles\/what-is-data-integrity-343068<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+Is+Data+Integrity%3F&rft.atitle=Technology+Networks&rft.aulast=Harmon%2C+C.&rft.au=Harmon%2C+C.&rft.date=20+November+2020&rft_id=https%3A%2F%2Fwww.technologynetworks.com%2Finformatics%2Farticles%2Fwhat-is-data-integrity-343068&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChrobakTheUS20-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChrobakTheUS20_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Chrobak, U. (17 August 2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.popsci.com\/story\/environment\/why-us-lose-power-storms\/\" target=\"_blank\">\"The US has more power outages than any other developed country. Here\u2019s why\"<\/a>. <i>Popular Science<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.popsci.com\/story\/environment\/why-us-lose-power-storms\/\" target=\"_blank\">https:\/\/www.popsci.com\/story\/environment\/why-us-lose-power-storms\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+US+has+more+power+outages+than+any+other+developed+country.+Here%E2%80%99s+why&rft.atitle=Popular+Science&rft.aulast=Chrobak%2C+U.&rft.au=Chrobak%2C+U.&rft.date=17+August+2020&rft_id=https%3A%2F%2Fwww.popsci.com%2Fstory%2Fenvironment%2Fwhy-us-lose-power-storms%2F&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TulsiGreater19-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TulsiGreater19_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Tulsi, B.B. (04 September 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labmanager.com\/business-management\/greater-awareness-and-vigilance-in-laboratory-data-security-776\" target=\"_blank\">\"Greater Awareness and Vigilance in Laboratory Data Security\"<\/a>. <i>Lab Manager<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.labmanager.com\/business-management\/greater-awareness-and-vigilance-in-laboratory-data-security-776\" target=\"_blank\">https:\/\/www.labmanager.com\/business-management\/greater-awareness-and-vigilance-in-laboratory-data-security-776<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Greater+Awareness+and+Vigilance+in+Laboratory+Data+Security&rft.atitle=Lab+Manager&rft.aulast=Tulsi%2C+B.B.&rft.au=Tulsi%2C+B.B.&rft.date=04+September+2019&rft_id=https%3A%2F%2Fwww.labmanager.com%2Fbusiness-management%2Fgreater-awareness-and-vigilance-in-laboratory-data-security-776&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RileyGitLab20-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RileyGitLab20_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Riley, D. (21 May 2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/siliconangle.com\/2020\/05\/21\/gitlab-runs-phishing-test-employees-20-handing-credentials\/\" target=\"_blank\">\"GitLab runs phishing test against employees \u2013 and 20% handed over credentials\"<\/a>. <i>Silicon Angle<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/siliconangle.com\/2020\/05\/21\/gitlab-runs-phishing-test-employees-20-handing-credentials\/\" target=\"_blank\">https:\/\/siliconangle.com\/2020\/05\/21\/gitlab-runs-phishing-test-employees-20-handing-credentials\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GitLab+runs+phishing+test+against+employees+%E2%80%93+and+20%25+handed+over+credentials&rft.atitle=Silicon+Angle&rft.aulast=Riley%2C+D.&rft.au=Riley%2C+D.&rft.date=21+May+2020&rft_id=https%3A%2F%2Fsiliconangle.com%2F2020%2F05%2F21%2Fgitlab-runs-phishing-test-employees-20-handing-credentials%2F&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FMSAutomated-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FMSAutomated_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.fms-inc.com\/sample-prep\/\" target=\"_blank\">\"Automated Sample Preparation\"<\/a>. Fluid Management Systems, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.fms-inc.com\/sample-prep\/\" target=\"_blank\">https:\/\/www.fms-inc.com\/sample-prep\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Automated+Sample+Preparation&rft.atitle=&rft.pub=Fluid+Management+Systems%2C+Inc&rft_id=https%3A%2F%2Fwww.fms-inc.com%2Fsample-prep%2F&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AgilentPAL-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AgilentPAL_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.agilent.com\/en\/product\/gas-chromatography\/gc-sample-preparation-introduction\/pal-auto-sampler-systems\" target=\"_blank\">\"PAL Auto Sampler Systems\"<\/a>. Agilent Technologies, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.agilent.com\/en\/product\/gas-chromatography\/gc-sample-preparation-introduction\/pal-auto-sampler-systems\" target=\"_blank\">https:\/\/www.agilent.com\/en\/product\/gas-chromatography\/gc-sample-preparation-introduction\/pal-auto-sampler-systems<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=PAL+Auto+Sampler+Systems&rft.atitle=&rft.pub=Agilent+Technologies%2C+Inc&rft_id=https%3A%2F%2Fwww.agilent.com%2Fen%2Fproduct%2Fgas-chromatography%2Fgc-sample-preparation-introduction%2Fpal-auto-sampler-systems&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BradshawTheImpo12-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BradshawTheImpo12_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bradshaw, J.T. (30 May 2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/d1wfu1xu79s6d2.cloudfront.net\/wp-content\/uploads\/2013\/10\/The-Importance-of-Liquid-Handling-Details-and-Their-Impact-on-Your-Assays.pdf\" target=\"_blank\">\"The Importance of Liquid Handling Details and Their Impact on Your Assays\"<\/a> (PDF). <i>European Lab Automation Conference 2012<\/i>. Artel, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/d1wfu1xu79s6d2.cloudfront.net\/wp-content\/uploads\/2013\/10\/The-Importance-of-Liquid-Handling-Details-and-Their-Impact-on-Your-Assays.pdf\" target=\"_blank\">https:\/\/d1wfu1xu79s6d2.cloudfront.net\/wp-content\/uploads\/2013\/10\/The-Importance-of-Liquid-Handling-Details-and-Their-Impact-on-Your-Assays.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Importance+of+Liquid+Handling+Details+and+Their+Impact+on+Your+Assays&rft.atitle=European+Lab+Automation+Conference+2012&rft.aulast=Bradshaw%2C+J.T.&rft.au=Bradshaw%2C+J.T.&rft.date=30+May+2012&rft.pub=Artel%2C+Inc&rft_id=https%3A%2F%2Fd1wfu1xu79s6d2.cloudfront.net%2Fwp-content%2Fuploads%2F2013%2F10%2FThe-Importance-of-Liquid-Handling-Details-and-Their-Impact-on-Your-Assays.pdf&rfr_id=info:sid\/en.wikipedia.org:LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n\n<!-- \nNewPP limit report\nCached time: 20210429194126\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.291 seconds\nReal time usage: 10.652 seconds\nPreprocessor visited node count: 10201\/1000000\nPreprocessor generated node count: 19993\/1000000\nPost\u2010expand include size: 69085\/2097152 bytes\nTemplate argument size: 28218\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 234.431 1 - -total\n 84.88% 198.982 2 - Template:Reflist\n 67.53% 158.313 15 - Template:Citation\/core\n 47.65% 111.708 11 - Template:Cite_web\n 24.16% 56.646 4 - Template:Cite_journal\n 8.13% 19.061 13 - Template:Efn\n 4.89% 11.468 23 - Template:Citation\/make_link\n 2.84% 6.664 3 - Template:Citation\/identifier\n 0.96% 2.244 6 - Template:Hide_in_print\n 0.76% 1.778 33 - Template:Clear\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12324-0!*!0!!en!5!* and timestamp 20210429194116 and revision id 41797\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering\">https:\/\/www.limswiki.org\/index.php\/LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","655f7d48a642e9b45533745af73f0d59_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f1\/Fig1_Liscouski_LabTechPlanMan20.png","https:\/\/upload.wikimedia.org\/wikipedia\/commons\/b\/ba\/PSM_V43_D075_Chemical_laboratory.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/56\/Fig3_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/ad\/Fig4_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/27\/Fig5_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/13\/Fig6_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/67\/Fig7_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/20\/Fig8_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3a\/Fig9_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/62\/Fig10_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/7a\/Fig11_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/80\/Fig12_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/42\/Fig13_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/41\/Fig14_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/12\/Fig15_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/44\/Fig16_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/59\/Fig17_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/02\/Fig18_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/5f\/Fig19_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8d\/Fig20_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a2\/Fig21_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8d\/Fig22_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/cd\/Fig23_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f0\/Fig24_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/23\/Fig25_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2b\/Fig26_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/56\/Fig27_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/db\/Fig28_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/91\/Fig29_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/00\/Fig30_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/25\/Fig31_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/25\/Fig32_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/fa\/Fig33_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2e\/Fig34_Liscouski_LabTechPlanMan20.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f5\/Fig35_Liscouski_LabTechPlanMan20.png"],"655f7d48a642e9b45533745af73f0d59_timestamp":1619725276,"0dd74698c3756484b9f52cadbcdc94dc_type":"article","0dd74698c3756484b9f52cadbcdc94dc_title":"Advanced engineering informatics: Philosophical and methodological foundations with examples from civil and construction engineering (Hartmann and Trappey 2020)","0dd74698c3756484b9f52cadbcdc94dc_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering","0dd74698c3756484b9f52cadbcdc94dc_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Advanced engineering informatics: Philosophical and methodological foundations with examples from civil and construction engineering\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nAdvanced engineering informatics: Philosophical and methodological foundations\r\nwith examples from civil and construction engineeringJournal\n \nDevelopments in the Built EnvironmentAuthor(s)\n \nHartmann, Timo; Trappey, AmyAuthor affiliation(s)\n \nTechnische Universit\u00e4t Berlin, National Tsing Hua UniversityPrimary contact\n \ntimo dot hartmann at tu-berlin dot deYear published\n \n2020Volume and issue\n \n4Article #\n \n100020DOI\n \n10.1016\/j.dibe.2020.100020ISSN\n \n2666-1659Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.sciencedirect.com\/science\/article\/pii\/S2666165920300168Download\n \nhttps:\/\/www.sciencedirect.com\/science\/article\/pii\/S2666165920300168\/pdfft (PDF)\n\nContents\n\n1 Abstract \n2 Introduction: Attempting to define advanced engineering informatics \n3 Knowledge representation and formalization \n\n3.1 Example 1: Formalizing engineering knowledge with ontology \n3.2 Example 2: Using logic to represent design knowledge \n3.3 Example 3: Optimization \n3.4 Example 4: Advanced data analytics \n\n\n4 Research methods \n\n4.1 Developing formalizations \n4.2 Verification and validation \n\n\n5 Discussion \n6 Conclusion \n7 Acknowledgements \n\n7.1 Competing interests \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nWe argue that the representation and formalization of complex engineering knowledge is the main aim of inquiries in the scientific field of advanced engineering informatics. We introduce ontology and logic as underlying methods to formalize knowledge. We also suggest that it is important to account for the purpose of engineers and the context they work in while representing and formalizing knowledge. Based on the concepts of ontology, logic, purpose, and context, we discuss different possible research methods and approaches that scholars can use to formalize complex engineering knowledge and to validate whether a specific formalization can support engineers with their complex tasks. On the grounds of this discussion, we suggest that research efforts in advanced engineering should be conducted in a bottom-up manner, closely involving engineering practitioners. We also suggest that researchers make use of social science methods while both eliciting knowledge to formalize and validating that formalized knowledge.\nKeywords: advanced engineering informatics, knowledge formalization, knowledge engineering, computing in engineering, research method, engineering\n\nIntroduction: Attempting to define advanced engineering informatics \nEngineers invent, design, analyze, build, test and maintain complex physical systems, structures, and materials to solve some of societies most urgent problems, but also to improve the quality of life of individuals. Engineering is artifact-centered and concerned with realizing physical products of all shapes, sizes, and functions. Engineers routinely use computers and engineering work is almost entirely digitized. Few tasks are conducted without some sort of digital support. Surprisingly still, some engineering disciplines, and in particular, civil engineers are termed (and term themselves) as digital laggards. Resistance to apply new digital technologies is high, and more often than not the real benefits of applying new digital technologies to support engineering design tasks is not perceived, visible, or existing.\nThe existing resistance towards adopting advanced computational tools has traditionally been attributed to individual and social characteristics of engineers themselves. For example, traditionally, studies focusing on the work of civil and construction engineers attributed resistance to the organizational characteristics of the industry, such as the seminal study of Mitropoulos and Tatum[1] about general industry characteristics, or the more recent study of Linderoth[2] looking at the specific collaboration network structure of the industry. Others like Davis and Songer[3] have attributed the resistance of engineers to adopt new technologies to individual characteristics of engineers, such as age, gender, general computer understanding, or experience.\nIndependent of resistance and its cause and despite the ever growing amount of digital applications that are used by engineers, it rather seems as if engineers are increasingly struggling with providing and improving our society\u2019s complex engineering systems.[4] This, in particular, holds in relation to the engineering systems within our built environment. Little research has provided insights into how the characteristics of computational tools influenced adoptions. Those studies that did showed that there seems to be a large difference between the general expectations of the engineers with the support that the tools could truly provide.[5][6] This paradox of supporting today\u2019s engineering work with adequate computational tools has triggered the engineering community to develop a new scientific field of study and inquiry: advanced engineering informatics.\nAdvanced engineering informatics is motivated by the quest to empower engineers to cope with the ever increasing complexity of the systems they have to provide. The discipline strives to provide means that allow engineers to leverage their understanding of the behavior of complex systems through advanced simulation and data analysis methods. It also strives at improving the collaboration and communication of engineers within the ever more complex collaborative interdisciplinary arrangements they face.\nUnlike other related disciplines, advanced engineering informatics focuses not on the automation of mundane tasks, but on developing, researching, and exploring methods to enhance the existing work environment of engineers. Advanced engineering informatics scholars believe that well-designed computational methods have the potential to empower engineers in ways that have previously not been possible. They believe that computers cannot only incrementally speed up engineering design work, but significantly disrupt engineering tasks throughout the entire product development life-cycle, from the early stages of conceptual design, to detailed engineering design, to production, to the maintenance of engineered systems.\nTo the above end, advanced engineering informatics acknowledges that engineering work is a knowledge-intensive activity.[7] Any research into how computational methods can support engineering work needs to start with an explicit formalization of the knowledge engineers posses. Advanced engineering informatics is a specific discipline of knowledge engineering[8] with an overarching research question: \u201cHow can we formalize complex engineering knowledge to develop advanced computational methods that help engineers to solve practical problems within their constraints and budgets?\u201d\nWith this research question\u2014above and beyond improving our understanding in how to formalize complex engineering knowledge through explicit representations and symbolic or numerical process models\u2014advanced engineering informatics is hence also concerned with understanding how such representations can support practical engineering work. To this end, topics for research are not only the development of advanced computational methods based on explicitly formulated knowledge, but also exploring the representation of information in graphical user interfaces, the provision of extensive knowledge bases through large scale databases, or how engineers and engineering groups can be supported in interpreting solutions and intermediate solution spaces.[7] In all of these endeavors, an explicit focus on engineering knowledge is required to advance this understanding.\nDespite the scientific and practical importance, most studies published in the scientific engineering journals fail to explicitly address aspects of engineering knowledge formalization and representation. This also holds for publications focusing on the engineering of our built environment. More often than not, new methods, algorithms, or results of data analysis efforts are presented without the contextualization of the suggested methods within a specific engineering context. Often it is not clear how suggested novel methods make use of explicitly formalized engineering knowledge and how the methods support engineers in their knowledge-intensive tasks. By large, the scientific engineering community still needs to establish a continuous growing body of scientific knowledge about how advanced computational methods can support engineers. Consequently, little general understanding about how novel computation methods can be implemented across tasks and engineering disciplines exists. This lack, in turn, has slowed down the development of solutions that could truly enhance practical engineering work.\n\n\nThis paper is an effort to refocus the current scientific discourse on the importance of engineering knowledge. To this end, we attempt to first provide a clear definition and description of the underlying philosophical basis of knowledge formulation and knowledge engineering as the foundation for all scientific inquiry within the field of advanced engineering informatics. We illustrate these definition and descriptions using a number of recently published articles that focus on the domain of built environment engineering as example.\nOur second goal for the paper is to start a discussion about the required methodological approaches for advanced engineering informatics research practice. So far there is little to no discourse about research methods within the field, which has significantly hindered its establishment among the other scientific disciplines. To catalyze this discourse, in the second part of the paper we suggest different research approaches and some underlying theories.\nOf course, like every other scientific discipline, the definitions, concepts, methods, and approaches associated with advanced engineering informatics are an ever-moving target. Therefore, this paper can only represent our current reflections and thinking in the field and is intended to provide food for thought and a catalyst for more reflective and vibrant discussion. By no means are the presented concepts of knowledge formalization and research methods meant as fixed bearing points, but rather as points of departure for wider theoretical explorations. Therefore, the paper also provides an elaborated discussion section with suggestions for future important areas of inquiry.\nIn the next section, we introduce the theoretical underpinnings of knowledge representation and knowledge formalization. That section also illustrates these underpinnings using four recently published research studies. Then different research methods that might be appropriate for advanced engineering informatics research are suggested. Finally, an extensive discussion with suggestions for important research directions are presented, along with conclusions.\n\nKnowledge representation and formalization \nSowa[8] defines knowledge engineering as the application of ontology and logic to the task of building computational models of some domain for some purpose. To inform advanced engineering informatics research, the definition provided in the introduction is informative as it focuses knowledge engineering on two important aspects. First, the definition suggests building computational models. Hence the definition proposes to move beyond the development of mathematical algorithms, towards models that already make computational prediction about a domain. Second, the definition also implies purpose, therefore requiring a focus on solving practical problems. These two aspects are of utmost importance for all research into advanced engineering informatics. The discipline is not concerned with conceiving new mathematical methods, algorithms, and calculation mechanisms, but rather it is concerned with using such basic computational methods to build models that compute tangible results that are relevant for a specific engineer. Furthermore, this relevance needs to be related to a practical engineering purpose within the wider product development cycle of an engineering system.\nMoreover, the definition points towards basic methodological approaches that advanced engineering informatics researchers have to be familiar with: ontology, logic, and computation. Within information science, an ontology is a formal representation of all a topic's concepts and their relations. An ontological knowledge representation is concerned with the knowledge of engineers about physical and abstract objects, relations between those objects, and events influencing those objects. Ontological representation allows for a commitment with respect to the model of the specific domain that is required as the basis for any computational method. With this commitment, ontologies help humans and computers understand and fully utilize domain knowledge. One important aspect of advanced engineering informatics research focuses on developing approaches for implementing computer-assisted engineering platforms that apply ontology-based theories and solutions.[9][10]\nEach ontology supporting such solutions needs to map the knowledge within a specific universe of discourse.[11] This universe of discourse should be a carefully bounded and focused micro-world[8] within an engineering discipline. Alternatively, it could also focus on a specific engineering collaboration between two engineering disciplines. To arrive at computational models as defined above, a bottom-up approach that focuses on a very specific engineering task is required. Moreover, domain ontology schema should be built and updated constantly together with all stakeholders of the knowledge domain. Knowledge is dynamically changing and growing, and, most importantly, it is possessed by multiple domain experts.[12]\nThe second methodological approach that is suggested by Sowa\u2019s definition is logic. Logic is the systematic study of inference that leads to the acceptance of a specific proposition. Such systematic studies require the clear formalization of a proposition and the development of a set of premises that may or may not support the conclusion. Logic as systematic study allows advanced engineering informatics researchers to formalize rules of inference that engineers use to arrive at conclusions, make decisions, or creatively develop design ideas.\nIn particular, the last point\u2014developing creative design ideas\u2014requires a thorough attention to logic. Currently, more often than not, the formalization of rules of inference can lead to logic that are to rigid or that focus on the formalization of irrelevant inference rules. In these cases, creative engineering design\u2014which is so important for improving complex engineered systems\u2014is inhibited.\nHowever, if applied well, logic allows the development of a theory of the intelligent reasoning approaches that engineers follow. Logic allows the formalization of complex engineering understanding about an engineering systems\u2019 behavior across space and time with respect to specific changes of the system under various specific environmental influences. Logic also allows the formalization of knowledge about important procedures that are required during production, or while maintaining an engineering system. Equally important to the formulation of knowledge about processes and procedures is that logic helps engineers account for specific constraints that bound such processes and procedures.\nBoth ontology and knowledge allow engineers to analyze complex engineering knowledge about the structure of an engineering system and its behavior, as well as procedures for its production and maintenance. However, ontology and logic by themselves do not yet allow for the description of engineering purpose. A classical example of this shortcoming is provided by Sowa[8], drawing upon Newton\u2019s second law of motion that relates force, mass, and acceleration. Newton\u2019s equation introduces an ontology that provides a clear and abstract description of the aspects related to the motion of an object. The formula also represents the logic of how force, mass, and acceleration are related. However, the formula itself does not yet propose how an engineer can use it to purposefully analyze a system. An engineer can use the law for three major purposes: to calculate mass from force and acceleration, force from mass and acceleration, or acceleration from mass and force. Which of these purposes is important for an engineer for a specific engineering task can only be formulated by representing the computation that is required within the specific context the engineer is in. Hence, purpose needs to be explicitly formulated while representing and formalizing engineering knowledge.\nNext to purpose, thought needs to specifically be applied towards the concept of \"context\" while formalizing engineering purpose. To a certain extent, it is impossible to define purpose without such attention to context. At the same time, however, it is important to consider context with respect to the knowledge formalized with ontology and logic. Both ontology and logic are models, and hence it is important to be explicit when and in which circumstances these models are applicable and when these might fail. Hence, understanding context is another important research activity within the field of advanced engineering informatics.\nIt is important for advanced engineering informatics scholars to consider that ontology, logic, and computation can only represent a very abstract model of the reasoning and knowledge of engineers. Formal knowledge representations are by nature fragmented and cannot get close to the true reasoning engineers use to come to their conclusions for specific tasks. No matter how fragmented and abstract ontologies and reasoning are, they, nevertheless, enable efficient communication, not only between engineers but also among advanced engineering informatics scientists.\nTo illustrate the above points, the following sub-sections describe how four recent studies suggested and validated four different computational methods for formalizing complex engineering knowledge within the area of built environment engineering. The examples have been identified as good practice examples by the two authors based on their experience as editors of the journal Advanced Engineering Informatics. It was not intended within the scope of this paper to provide a structural literature review, but rather to illustrate the above concepts with a number of loosely selected previous research studies.\n\nExample 1: Formalizing engineering knowledge with ontology \nThe objective of developing formal ontologies is to help humans and computers understand and, hence, fully utilize domain knowledge in various knowledge management systems. Domain ontology schema should be built and updated constantly as a collective intelligence, since knowledge is considered dynamically changing and growing and, most importantly, can be contributed by multiple domain experts.[12]\nAn example for such a system is Yuan et al.\u2018s effort to model the residual value risk around the vulnerability of infrastructure projects.[13] Financial responsibility on these projects is shared by public and private parties. Understanding financial risks that occur during the delivery life-cycle of such projects is important. Estimating these risks is a complex task that engineers are concerned with already during the conceptual design stages, and that is crucial to thoroughly draft contractual agreements between the public and private partners involved in such projects.\nYuan et al. formalized the engineering knowledge of this specific domain by proposing an ontology represeting risk sources, risk events, risk consequences, exposures, resilience factors, and contextual sensitivity characteristics that might influence the risks of a specific project. The study also instantiated the ontology formalizing the specific knowledge of an illustrative bridge project and validated the ontology by conducting a survey among domain experts.\nThe study shows the utility of formalizing knowledge using ontologies. The authors illustrate how the ontology allows to visualize the risk factors using knowledge graphs and how these visualizations helped to estimate the financial risks of a project. The study also illustrates how the formal representation of the knowledge allows the computation of automated reasoning paths, for example, to understand the effect of design or environmental changes on a specific risk profile.\n\nExample 2: Using logic to represent design knowledge \nAn example of how to use logic to formalize engineering knowledge can be found in Min et al.\u2018s study that developed rule-based patterns for laying out theme parks.[14] Designing leisure spaces in a theme park is a highly knowledge-intensive activity. Theme parks need to provide a highly complex and multi-layered service environment to satisfy visitors. In their study, Min et al. identified and formalized patterns used in a number of successful theme parks and combined them into a reasoning system.\nSome logical patterns formalized in the study are, for example, that facilities such as attractions, restaurants, and shops are equally distributed around a park\u2019s centroid. Another logical pattern Min et al. identified and formalized is that building entrances are located at pathways that exhibit relatively low traffic. The authors also illustrated how these patterns can be used by developing a software implementation for theme park design and applying the software to design a new theme park in South Korea. The logic was validated by interviewing experts and by conducting design experiments with four experienced experts.\n\nExample 3: Optimization \nMuch work within the field of advanced engineering informatics has focused on how design optimization can support engineers to identify optimal designs among a set of alternatives. During design optimization, ontology and logic play an important role, as it is required to devise a mathematical formulation of the design problem. To develop this formulation, researchers have to identify variables that describe the alternatives and then relate these variables logically within an objective function that is to be maximized or minimized. Additionally, a number of constraints have to be logically formulated based on the initial design variables. If design problems can be formulated adequately, a large number of computational optimization methods are available that can be applied. While the development of new optimization algorithms would rather fall within the domain of computer science or mathematics, the formulation of design optimization problems is an important topic of advanced engineering informatics research.\nAn example of research that formulates a design optimization problem around a complex engineering task is Jin et al.\u2018s study into how to formulate the planning of scaffolding required for complex piping installation work.[15] Designing the best set-up for scaffolding is a complex engineering task because of the spatial relationship between the locations in which work needs to be supported and the requirements to set-up supporting structures.\nFor formulating the optimization problem, Jin et al. developed a rule-based logic of scaffolding placement and linked these rules with an explicit ontological description combining the timing of construction work tasks, the location of these tasks, and of the pipes\u2019 geometries. The formulation also included different possible postures that workers can be in to install a pipe in an attempt to allow for the optimization of ideal working postures for productive installation work. Different constraints were also formulated such as minimum and maximum acceptable heights for working conditions to bound the automated alternative generation.\nThe formulation of the optimization problem was validated on a practical example concerned with the installation of 71 different pipes for a 21-meter-high industrial plant. This illustrative validation could vividly show how a knowledge representation can help site engineers to set up optimal scaffolding that minimizes the amount of pipes that cannot be installed with a specific set-up while maximizing the productivity of installation work.\n\nExample 4: Advanced data analytics \nSimilar to optimization, the last two decades have seen a large amount of studies that applied advanced data analytics methods, in particular neural network-based machine learning, in an effort to develop methods for supporting engineers. Similar to optimization studies, it is important to focus explicitly on the representation of complex engineering knowledge that is inherent to data analytics studies in order to make those studies meaningful to engineering practice. From a knowledge representation perspective, machine learning methods transform collected data input that engineers can not easily interpret to an output that is interpretable to engineers.[8] Advanced engineering informatics research studies these aspects of knowledge representation around the application of well-known data analytics methods from computer science and mathematics. Advanced engineering informatics also explores how such knowledge representations and translations can help engineers to deal with their complex engineering tasks.\nAn example of such a study is the work by Leng et al.[16] that developed a forecasting method for wind power systems. In their work, the authors suggest a method to translate wind power signals that are hard to interpret by applying the ridgelet transform method, which allows to mathematically model the singular changes within the wind signal more accurately. Using the wind signals as input features, with the ridgelet transform adjusted, a neural network can be trained to forecast wind power accurately.\nThe authors then demonstrate the utility of the suggested forecasting method by applying it to a wind farm in Alberta. The authors were able to illustrate how the method allows interpretable outputs forecasting wind power for different yearly seasons and even specific days. These predictions can then be used by wind farm engineers to not only design better wind farm layouts, but also to improve the maintenance and management of wind farms in operation.\n\nResearch methods \nAfter providing a philosophical basis for research within the field of advanced engineering informatics through focusing the discussion on ontology, logic, purpose, and context, this section will provide some fundamental propositions for how to methodologically approach scientific research. To date, there has been little discussion about research methods in the field, and clear conventions for how to systematically approach research questions have been largely missing. Scientific methods should be empirical in their approach to acquiring knowledge, and they should allow researchers to apply skepticism about what is empirically observed. With this in mind, the section is structured around the two major research tasks of first developing knowledge representations and secondly how to verify, validate, and illustrate them.\n\nDeveloping formalizations \nOne of the major research tasks of advanced engineering informatics is the formalization of knowledge with ontology and logic and to represent it. For formalization, well-established methods exist and are widely used. For ontologies, researchers often generate knowledge maps that represent the different identified concepts and that can be created with ontological editors, such as the widely used Prot\u00e9g\u00e9.[17] Logic is usually represented in mathematical notations, algorithms in pseudo code, or process diagrams. Metamodels, such as the ones listed above, for representing formalizing models of engineering knowledge are in widespread use within the research community.\nHowever, little thought has so far been given towards how to systematically elicit complex engineering knowledge from experts and engineering practice. By large, most of the knowledge formalized represents the knowledge of the researchers themselves. Pragmatically, this approach has helped the field to develop because the researchers are often also expert engineers. Scientifically, however, this practice is challenging as it lacks the empirical and systematic character that is asked for by sound scientific practice. Only a few specific methods have been proposed or utilized to elicit engineering knowledge from practitioners.[18][19][20] What is common among the few suggested methods is that they either rely upon social science-based methods or on the formal analysis of engineering documents and models.\nSocial science-based methods that have been suggested in the past were based on interviews with engineers and the organization of workshops and focus groups, as well as more experimental methods. One of the more advanced methods proposed is the use of so-called design charrettes. (Senescu and Haymaker, 2013) Design charrettes propose to capture knowledge of engineers by organizing short, intensive design exercises with professionals that can be observed. An analytical analysis of these observations then allow researchers to better understand the knowledge the engineers applied during the charrette for the consecutive structural formalization. As another example of a social science approach, Hartmann et al. proposed the ethnographic action research method.[18] Ethnographic action research suggests that the researcher gets immersed in engineering work settings by working alongside practicing engineers. They suggest that this immersion allows the researcher to gain the necessary deep insights about practical engineering knowledge prior to formalization.[18]\nNext to social science research methods, researchers have relied on the structured analysis of documents and other artifacts. As discussed earlier, Min et al.[14], for example, have formalized design patterns for the design of theme parks. The study identified the patterns through a structured analysis of design documents existing for different realized theme parks.\nWith the advancement of text mining and pattern recognition techniques, recently many studies have been published that use automated methods to extract information from existing design documents and models. This field has developed into a vibrant research area. Automated text mining methods have been applied to formalize engineering knowledge from patents[21][22], identify research trends for building energy savings[23], analyze construction site accidents[24], predict construction cost overruns[25], retrieve CAD drawings[26], or extract best practices from simulation modeling guidelines.[27] Additionally, several graph-based pattern mining methods have been applied to architectural floor plan design[28] or to the automated extraction and formalization of construction process patterns.[29] Finally, geometric pattern recognition techniques have also been developed to support the aerodynamic design of vehicles.[30]\nConsidering that engineers have compiled an extensive digital collection of such design documents, we expect that the years to come will see a further acceleration of this area of research. In practice, the results could provide engineers with dedicated domain-specific search engines that will allow them to better find and understand previous design solutions that are adaptable to a design task at hand. Moreover, the identification of such patterns can lead to an increasing number of evidence-based design tools to support a wide range of engineering tasks. Finally, test mining and pattern matching might allow engineers to derive new insights into the behavior of engineering products and materials, as it will allow the combination of a large amount of previous measurements and test results.\n\nVerification and validation \nNext to the question of how to best elicit and formalize knowledge, another question for sound scientific research in the field of advanced engineering informatics is how to ensure that a proposed knowledge formulation is appropriate and useful. Sound scientific practice requires that a knowledge representation is systematically verified and validated. This section will summarize some of the most common approaches used for verification and validation so far.\nFor ontologies, verification is the process of ensuring that the axioms of the ontology reflect the intentions of the author.[31] Building ontologies is an error-prone activity, and it is very difficult to structure ontologies so that they do not allow for unintended inferences, for example, through the introduction of unsatifiable axioms. Ontologies can be verified with the built-in reasoners within common ontology development tools, such as the earlier introduced Prot\u00e9g\u00e9[17], which can automatically detect defects in the ontology. While developing ontologies, these automatic reasoners should be used frequently to avoid the propagation of systematic errors early on. As an ontology is growing and evolving, fixing such systematic errors will get increasingly difficult and time-consuming.\nVerifying a computational method that is based on logic is a more difficult problem. Strictly speaking, sound verification would require mathematically proving the computation. Finding mathematical proofs, however, quickly turns too complex even for relatively simple computations. Another option is a combinatorical approach that controls the outputs of every possible input within the context of the computation. Again, however, true combinatorical efforts are not feasible in real-world research settings, even if the context in which a specific computation should work is carefully bounded. A solution to this problem is the application of sensitivity analysis methods in relation to well defined sampling strategies for different possible combinations of the input values.[32] More often than not, however, most studies still simply rely on the use of illustrative examples to verify their suggested computational methods. While such an approach is widely accepted, researchers should at a minimum provide a sound argument for the appropriateness of the used illustrative example in relation to the complexity of the real world engineering challenge at hand.\nIn contrast to verification, validation is the process of evaluating whether the knowledge representation is fit to the engineering purpose at hand. Evaluating the validity requires a closer attention to practical engineering than verification itself and is often more difficult to conduct. While a sound verification only needs to closely evaluate the internal structure of a knowledge representation, validation requires to evaluate a representation within its context and for its specific purpose. Because of this requirement, it is much harder to provide convincing evidence for the validity of a solution, and researchers need to very carefully plan and conduct validation exercises.\nValidation can be approached at different levels. By far the most often conducted validation and also the most easiest is to apply the knowledge representation to an illustrative example of an engineering task. In this way, it is possible to show that the representation fits its purpose; however, oftentimes such a validation is not very convincing. At a minimum, it is important that the illustrative example used for the validation represents a challenge that engineers would encounter in practice. Far too often, however, studies use radically simplified examples that do not reflect the complexity of a problem that engineers would face in practice. While simplified illustrative examples are a great vehicle for verification, such efforts can hardly be considered actual validation.\nOther approaches that can be used to validate knowledge representation systems center around the implementation of a prototype of a computational method that implements the representation. Simple efforts use such a prototype to establish a demonstration of the system at work and then ask engineers for their opinion on the system. Again, more often than not, such efforts to validation are not very convincing. The methodological problem with the approach is that it is hard to develop a good sampling strategy to target engineers that reflect different levels of expertise and backgrounds that would be required to be able to generalize the results. Moreover, even if a sound sampling strategy can be developed it is very hard to ensure participation of engineers in such studies. Some researchers, therefore, have reverted to student surveys, which are even less convincing.\nA slightly more advanced approach to validation is to ask engineers or engineering students to use the prototype for solving an engineering task. Such approaches can provide much stronger evidence, but any prototype also needs to be designed very carefully. For one, there often is the tendency to over-structure the experimental setting, giving the test subjects an overly targeted assignment for using the prototype. Moreover, some studies have then compared such efforts with a control group of engineers that have solved the same task without the support of the prototype. While seemingly providing strong validation for the working of the prototype, such efforts can hardly show the utility of the knowledge representation to empower engineers to cope with an engineering problem in a significantly different manner than was possible before.\nA less structured effort to validation that centers around devising open-ended experimental settings in which engineers solve a design challenge without too much structuring of the process often proves to be more convincing. Such open-ended design experiments resemble the earlier introduced design charette studies, albeit this time the designers use a prototypical implementation for the design task at hand. While it is possible to ask participants in these studies questions before and after the experiment, the experiment itself should be closely observed and ideally recorded. The observations and recordings can then be analyzed using qualitative research methods that are well established in the social sciences.[33][34] Such qualitative data analysis methods can often provide very strong evidence that the prototype empowered engineers in the design experiments to work in a substantially different manner than in current practice. Again, such experiments can be conducted with students as well, but, of course, it is more convincing if practicing engineers can be convinced to participate.\nBy far the most convincing validation is to show that a prototype or even an already more developed system design based upon a structured knowledge representation can support practicing engineers. Unfortunately, collecting such evidence is seldom feasible as engineers need to be found that are willing and able to use a new tool in practice. The ethnographic action research method[18] can serve as a starting point for slowly convincing and training practitioners by working with them for a prolonged period. While engineers work on a task, the researcher can shadow the work of the engineers by implementing observed decision making and design activities using a prototypical system. In this manner, evidence for the utility of the prototype can be built up step-by-step, and engineers can be convinced little by little to implement the system directly. Complicating the situation for the researcher, all these efforts need to be closely monitored by structurally collecting data and documenting evidence. Again qualitative data collection and analysis methods from the social sciences[33][34] can help with such endeavors.\n\nDiscussion \nThis paper has developed a philosophical foundation for establishing a more coherent field of advanced engineering informatics that is solidly based upon formalizing complex engineering knowledge. This foundation suggests that ontology and logic form the basis for scientific knowledge formalization efforts in the field, and associated research needs to be concerned with purpose and context. The paper also suggests a number of research approaches that can help to elicit knowledge from engineers for formalization and help researchers to design studies for the verification and validation of such formulations. All of the above discussions and examples do have a focus on built environment engineering, a discipline that provides some of the most complex engineering systems to date.\nThe suggested approaches are cumbersome, particularly since engineers in practice posses an enormous amount of background knowledge, have multiple often conflicting purposes, and work in a myriad of differing, complex settings. Even worse, on top of this highly heterogeneous character of knowledge, that knowledge is also fluid and ever-changing.\nTo cope with these difficulties, we suggest that scientific research in the field is approached bottom-up, paying detailed attention to the specific context and purpose of engineers. Knowledge can only be explored in small chunks that correspond to very specific engineering purposes and contexts. Advanced engineering informatics must therefore be an ever-evolving research field that is advanced in small steps. In turn, generalizations, definitions, and anticipations will always be inconsistent. Conditions observed within a specific context might be abnormal for other contexts, developed solutions for specific purposes might be conflicting in other contexts, and applications might have unanticipated outcomes that can only be recognized much later.\nIn all, it is unlikely that scientists working in the field of advanced engineering informatics will discover great breakthroughs. Advanced engineering informatics is a humble research discipline that carefully needs to be built up slowly with a growing body of knowledge that is continuously challenged, criticized and revised. Studies need to be designed that carefully build upon each other. Each single research project, PhD thesis, or scientific publication can only contribute a little chunk of knowledge to the advancement of the field.\nWith this in mind, the field also needs to carefully review its current practice in the sense of whether studies rigorously and explicitly built upon previous work. It is often far too easy to start a study from scratch, develop a new computational method, and claim its utility by illustrating its use on some self-proclaimed engineering problem that is to be solved. Such studies, however, lack the required scientific rigor in providing empirical evidence and do little to move the field systematically forward. Moreover, such studies often fail to empower engineers with methods to cope with the ever increasing complexity of engineering systems they need to design, produce, and maintain.\nTo provide true scientific stepping stones that advance our knowledge, researchers need to thoroughly understand the body of research that has been conducted previously. Literature reviews that inform studies should both carefully summarize work that has been done to support similar engineering purposes, for example, by thoroughly understanding engineering disciplines, tasks, and contexts that have been explored in previous studies. At the same time, researchers have to develop a deep understanding about how the specific set of computational methods that they intend to apply for formalizing knowledge has been used to solve engineering problems in a wide range of other contexts. Literature reviews for advanced engineering informatics studies, therefore, always need to be twofold evaluating the state of the art in supporting a specific engineering context and evaluating the state of the art of a specific computational method. Only then will a consistent academic stream of inquiry across time and space emerge that allows to slowly generalize findings to answer the two main research questions of the field: \u201cHow do we best formalize complex engineering knowledge with novel computational methods?\u201d and \u201cHow can engineers be empowered by computational methods to significantly improve their work?\u201d\nAnother important aspect is that reproductive studies that further verify and validate previous studies are important to advance knowledge. Again, far too often researchers choose not to conduct reproductive studies or truthfully frame their conducted work as reproductive. The lure of being able to claim a significant scientific breakthrough is often too tempting. In the defense of the scientists, far too often, reviewers of scientific work also fail to acknowledge the important aspect of reproductive studies. To become a more coherent field, it is important that attitudes change. Studies that can replicate earlier findings, even in close and similar engineering contexts, should be considered as important. These studies can increase the sophistication of validation methods, provide supporting or debunking evidence, and further discuss insights using additional empirical evidence collected. The field should consider such studies as equally important for advancing our knowledge than studies that unconvincingly claim the utility of newly conceived methods.\nDespite all these problems and the cumbersome suggested research that is required to formalize engineering knowledge, the clear communication that formal knowledge representation allows for will allow others to continuously improve the utility of a specific computational approach in empowering engineers in ways that have previously not been possible. Additionally, clear communication will allow for the development of shared models for achieving the integration of different engineering disciplines, which are required to design, produce, and maintain our ever more complex engineered systems. Finally, such an explicit focus on knowledge representation with ontology and logic, together with a specific focus on purpose and context, will allow for the generalized application of different methods across engineering fields and hence further establish the scientific discipline of advanced engineering informatics.\n\nConclusion \nTo refocus current research efforts in developing computational methods within the wider field of engineering, and in particular with respect to built environment engineering, we argue that knowledge representation is the main research effort that is required to develop technologies that not only automate mundane engineering tasks, but also provide engineers with tools that will allow them to do things they were not able to do before. We suggest that such tools will be required so that engineers can deal with the ever increasing complexity of the modern engineering systems they need to deliver.\nTo focus scientific work in the field of advanced engineering informatics on knowledge representation, we first introduce the underlying philosophical concepts of knowledge representation and formalization. To this end, we heavily draw on the seminal work of Sowa.[8] We then illustrate these concepts using four recently published studies. Based on the theoretical concepts, we share our thoughts about possible research methods that scholars can draw upon while developing and empirically validating knowledge representations. The suggested research methods are meant to start an ongoing discussion about how to best conduct research in the field of advanced engineering informatics.\nIn conclusion, we hope that this position paper can help scientists to understand the field of advanced engineering informatics and its importance better. We also hope that the paper can support scholars in designing studies within the field that can improve our knowledge of how best to use computational methods to formalize complex engineering knowledge. As a follow-up step to this paper, we suggest that researchers conduct a structured and comprehensive literature review to further expound upon the field of advanced engineering informatics.\n\nAcknowledgements \nCompeting interests \nThe authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.\n\nReferences \n\n\n\u2191 Mitropoulos, P.; Tatum, C.B. (2000). \"Forces Driving Adoption of New Information Technologies\". Journal of Construction Engineering and Management 126 (5): 340\u20138. doi:10.1061\/(ASCE)0733-9364(2000)126:5(340).   \n\n\u2191 Linderoth, H.C.J. (2010). \"Understanding adoption and use of BIM as the creation of actor networks\". Automation in Construction 19 (1): 66\u201372. doi:10.1016\/j.autcon.2009.09.003.   \n\n\u2191 Davis, K.A.; Songer, A.D. (2009). \"Resistance to IT Change in the AEC Industry: Are the Stereotypes True?\". Journal of Construction Engineering and Management 135 (12): 1324-1333. doi:10.1061\/(ASCE)CO.1943-7862.0000108.   \n\n\u2191 de Weck, O.L.; Roos, D.; Magee, C.L. (2011). Engineering Systems: Meeting Human Needs in a Complex Technological World. MIT Press. ISBN 9780262016704.   \n\n\u2191 Hartmann, T.; van Meerveld, H.; Vossebeld, N. et al. (2012). \"Aligning building information model tools and construction management methods\". Automation in Construction 22 (1): 605\u201313. doi:10.1016\/j.autcon.2011.12.011.   \n\n\u2191 Hartmann, T. (2011). \"Goal and Process Alignment during the Implementation of Decision Support Systems by Project Teams\". Journal of Construction Engineering and Management 137 (12): 1134\u201341. doi:10.1061\/(ASCE)CO.1943-7862.0000389.   \n\n\u2191 7.0 7.1 Kunz, J.C.; Smith, I.F.C.; Tomiyama, T. (2002). \"Editorial\". Advanced Engineering Informatics 16 (1): 1\u20132. doi:10.1016\/S1474-0346(02)00004-6.   \n\n\u2191 8.0 8.1 8.2 8.3 8.4 8.5 Sowa, J.F. (2014). Principles of Semantic Networks: Explorations in the Representation of Knowledge. Elsevier. ISBN 9781483221144.   \n\n\u2191 Kotis, K.; Papasalourou, A.; Vouros, G. et al. (2011). \"Enhancing the Collective Knowledge for the Engineering of Ontologies in Open and Socially Constructed Learning Spaces\". Journal of Universal Computer Science 17 (12): 1710\u201342. doi:10.3217\/jucs-017-12-1710.   \n\n\u2191 Huang, C.J.; Trappey, A.J.C.; Wu, C.Y. (2008). \"Develop a Formal Ontology Engineering Methodology for Technical Knowledge Definition in R&D Knowledge Management\". In Curran, R.; Chou, S.Y.; Trappey, A.. Collaborative Product and Service Life Cycle Management for a Sustainable World. Springer. pp. 495\u2013502. doi:10.1007\/978-1-84800-972-1_46. ISBN 9781848009721.   \n\n\u2191 Hartmann, T.; Amor, R.; East, E.W. (2017). \"Information Model Purposes in Building and Facility Design\". Journal of Computing in Civil Engineering 31 (6): 04017054. doi:10.1061\/(ASCE)CP.1943-5487.0000706.   \n\n\u2191 12.0 12.1 Valarakos, A.G.; Karkaletsis, V.; Alexopoulou, D. et al. (2006). \"Building an allergens ontology and maintaining it using machine learning techniques\". Computers in Biology and Medicine 36 (10): 1155-1184. doi:10.1016\/j.compbiomed.2005.09.007.   \n\n\u2191 Yuan, J.; Li, X.; Chen, K. et al. (2018). \"Modelling residual value risk through ontology to address vulnerability of PPP project system\". Advanced Engineering Informatics 38: 776-793. doi:10.1016\/j.aei.2018.10.009.   \n\n\u2191 14.0 14.1 Min, D.A.; Hyun, K.H.; Kim, S.-J. et al. (2017). \"A rule-based servicescape design support system from the design patterns of theme parks\". Advanced Engineering Informatics 32: 77\u201391. doi:10.1016\/j.aei.2017.01.005.   \n\n\u2191 Jin, H.; Nahangi, M.; Goodrum, P.M. et al. (2017). \"Model-based space planning for temporary structures using simulation-based multi-objective programming\". Advanced Engineering Informatics 33: 164\u201380. doi:10.1016\/j.aei.2017.07.001.   \n\n\u2191 Leng, H.; Li, X.; Zhu, J. et al. (2018). \"A new wind power prediction method based on ridgelet transforms, hybrid feature selection and closed-loop forecasting\". Advanced Engineering Informatics 36: 20\u201330. doi:10.1016\/j.aei.2018.02.006.   \n\n\u2191 17.0 17.1 Gennari, J.H.; Musen, M.A.; Fergerson, R.W. et al. (2003). \"The evolution of Prot\u00e9g\u00e9: An environment for knowledge-based systems development\". International Journal of Human-Computer Studies 58 (1): 89\u2013123. doi:10.1016\/S1071-5819(02)00127-1.   \n\n\u2191 18.0 18.1 18.2 18.3 Hartmann, T.; Fischer, M.; Haymaker, J. (2009). \"Implementing information systems with project teams using ethnographic\u2013action research\". Advanced Engineering Informatics 23 (1): 57\u201367. doi:10.1016\/j.aei.2008.06.006.   \n\n\u2191 Senescu, R.R.; Haymaker, J.R. (2013). \"Evaluating and improving the effectiveness and efficiency of design process communication\". Advanced Engineering Informatics 27 (2): 299\u2013313. doi:10.1016\/j.aei.2013.01.003.   \n\n\u2191 Jung, N.; Lee, G. (2019). \"Automated classification of building information modeling (BIM) case studies by BIM use based on natural language processing (NLP) and unsupervised learning\". Advanced Engineering Informatics 41: 100917. doi:10.1016\/j.aei.2019.04.007.   \n\n\u2191 Govindarajan, U.H.; Trappey, A.J.C.; Trappey, C.V. (2019). \"Intelligent collaborative patent mining using excessive topic generation\". Advanced Engineering Informatics 42: 100955. doi:10.1016\/j.aei.2019.100955.   \n\n\u2191 Wang, J.; Chen, Y.-J. (2019). \"A novelty detection patent mining approach for analyzing technological opportunities\". Advanced Engineering Informatics 42: 100941. doi:10.1016\/j.aei.2019.100941.   \n\n\u2191 Ding, Z.; Li, Z.; Fan, C. (2018). \"Building energy savings: Analysis of research trends based on text mining\". Automation in Construction 96: 398\u2013410. doi:10.1016\/j.autcon.2018.10.008.   \n\n\u2191 Zhang, F.; Fleyeh, H.; Wang, X. et al. (2019). \"Construction site accident analysis using text mining and natural language processing techniques\". Automation in Construction 99: 238\u201348. doi:10.1016\/j.autcon.2018.12.016.   \n\n\u2191 Williams, T.P.; Gong, J. (2014). \"Predicting construction cost overruns using text mining, numerical data and ensemble classifiers\". Automation in Construction 43: 23\u201329. doi:10.1016\/j.autcon.2014.02.014.   \n\n\u2191 Yu, W.-D.; Hsu, J.-Y. (2013). \"Content-based text mining technique for retrieval of CAD documents\". Automation in Construction 31: 65\u201374. doi:10.1016\/j.autcon.2012.11.037.   \n\n\u2191 Kestel, P.; K\u00fcgler, P.; Zirngibl, C. et al. (2019). \"Ontology-based approach for the provision of simulation knowledge acquired by Data and Text Mining processes\". Advanced Engineering Informatics 39: 292\u2013305. doi:10.1016\/j.aei.2019.02.001.   \n\n\u2191 Strug, B.; \u015alusarczyk, G. (2009). \"Reasoning about designs through frequent patterns mining\". Advanced Engineering Informatics 23 (4): 361\u201369. doi:10.1016\/j.aei.2009.06.007.   \n\n\u2191 Sigalov, K.; K\u00f6nig, M. (2017). \"Recognition of process patterns for BIM-based construction schedules\". Advanced Engineering Informatics 33: 456\u201372. doi:10.1016\/j.aei.2016.12.003.   \n\n\u2191 Graening, L.; Sendhoff, B. (2014). \"Shape mining: A holistic data mining approach for engineering design\". Advanced Engineering Informatics 28 (2): 166\u201385. doi:10.1016\/j.aei.2014.03.002.   \n\n\u2191 Matentzoglu, N.; Vigo, M.; Jay, C. etc. (2018). \"Inference Inspector: Improving the verification of ontology authoring actions\". Journal of Web Semantics 49: 1\u201315. doi:10.1016\/j.websem.2017.09.004.   \n\n\u2191 Saltelli, A.; Chan, K.; Scott, E.M., ed. (2000). Sensitivity Analysis. Series in Probability and Statistics. Wiley. ISBN 9780471998921.   \n\n\u2191 33.0 33.1 Miles, M.B.; Huberman, A.M. (1994). Qualitative Data Analysis: An Expanded Sourcebook. SAGE Publications. ISBN 9780803955400.   \n\n\u2191 34.0 34.1 Corbin, J.; Strauss, A. (2015). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (4th ed.). SAGE Publishing. ISBN 9781412997461.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\">https:\/\/www.limswiki.org\/index.php\/Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on engineering informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 5 January 2021, at 22:57.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 419 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","0dd74698c3756484b9f52cadbcdc94dc_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Advanced_engineering_informatics_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Advanced engineering informatics: Philosophical and methodological foundations with examples from civil and construction engineering<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>We argue that the representation and formalization of complex engineering knowledge is the main aim of inquiries in the scientific field of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Engineering_informatics\" class=\"extiw wiki-link\" title=\"wikipedia:Engineering informatics\" data-key=\"b9306296db1f0d2935eda0c125b63cd9\">advanced engineering informatics<\/a>. We introduce <a href=\"https:\/\/www.limswiki.org\/index.php\/Ontology_(information_science)\" title=\"Ontology (information science)\" class=\"wiki-link\" data-key=\"52d0664bde4b458e81fbc128b911a4a6\">ontology<\/a> and logic as underlying methods to formalize <a href=\"https:\/\/www.limswiki.org\/index.php\/Information#As_an_influence_which_leads_to_a_transformation\" title=\"Information\" class=\"wiki-link\" data-key=\"33bf2abbc89b17a3f4ca7bd13bce9e78\">knowledge<\/a>. We also suggest that it is important to account for the purpose of engineers and the context they work in while representing and formalizing knowledge. Based on the concepts of ontology, logic, purpose, and context, we discuss different possible research methods and approaches that scholars can use to formalize complex engineering knowledge and to validate whether a specific formalization can support engineers with their complex tasks. On the grounds of this discussion, we suggest that research efforts in advanced engineering should be conducted in a bottom-up manner, closely involving engineering practitioners. We also suggest that researchers make use of social science methods while both eliciting knowledge to formalize and validating that formalized knowledge.\n<\/p><p><b>Keywords<\/b>: advanced engineering informatics, knowledge formalization, knowledge engineering, computing in engineering, research method, engineering\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction:_Attempting_to_define_advanced_engineering_informatics\">Introduction: Attempting to define advanced engineering informatics<\/span><\/h2>\n<p>Engineers invent, design, analyze, build, test and maintain complex physical systems, structures, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Materials_informatics\" title=\"Materials informatics\" class=\"wiki-link\" data-key=\"4dd2125beb9794d0a679b921981f1ddc\">materials<\/a> to solve some of societies most urgent problems, but also to improve the quality of life of individuals. Engineering is artifact-centered and concerned with realizing physical products of all shapes, sizes, and functions. Engineers routinely use computers and engineering work is almost entirely digitized. Few tasks are conducted without some sort of digital support. Surprisingly still, some engineering disciplines, and in particular, civil engineers are termed (and term themselves) as digital laggards. Resistance to apply new digital technologies is high, and more often than not the real benefits of applying new digital technologies to support engineering design tasks is not perceived, visible, or existing.\n<\/p><p>The existing resistance towards adopting advanced computational tools has traditionally been attributed to individual and social characteristics of engineers themselves. For example, traditionally, studies focusing on the work of civil and construction engineers attributed resistance to the organizational characteristics of the industry, such as the seminal study of Mitropoulos and Tatum<sup id=\"rdp-ebb-cite_ref-MitropoulosForces00_1-0\" class=\"reference\"><a href=\"#cite_note-MitropoulosForces00-1\">[1]<\/a><\/sup> about general industry characteristics, or the more recent study of Linderoth<sup id=\"rdp-ebb-cite_ref-LinderothUnder10_2-0\" class=\"reference\"><a href=\"#cite_note-LinderothUnder10-2\">[2]<\/a><\/sup> looking at the specific collaboration network structure of the industry. Others like Davis and Songer<sup id=\"rdp-ebb-cite_ref-DavisResist09_3-0\" class=\"reference\"><a href=\"#cite_note-DavisResist09-3\">[3]<\/a><\/sup> have attributed the resistance of engineers to adopt new technologies to individual characteristics of engineers, such as age, gender, general computer understanding, or experience.\n<\/p><p>Independent of resistance and its cause and despite the ever growing amount of digital applications that are used by engineers, it rather seems as if engineers are increasingly struggling with providing and improving our society\u2019s complex engineering systems.<sup id=\"rdp-ebb-cite_ref-DeWeckEngineering11_4-0\" class=\"reference\"><a href=\"#cite_note-DeWeckEngineering11-4\">[4]<\/a><\/sup> This, in particular, holds in relation to the engineering systems within our built environment. Little research has provided insights into how the characteristics of computational tools influenced adoptions. Those studies that did showed that there seems to be a large difference between the general expectations of the engineers with the support that the tools could truly provide.<sup id=\"rdp-ebb-cite_ref-HartmannAlign12_5-0\" class=\"reference\"><a href=\"#cite_note-HartmannAlign12-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HartmannGoal11_6-0\" class=\"reference\"><a href=\"#cite_note-HartmannGoal11-6\">[6]<\/a><\/sup> This paradox of supporting today\u2019s engineering work with adequate computational tools has triggered the engineering community to develop a new scientific field of study and inquiry: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Engineering_informatics\" class=\"extiw wiki-link\" title=\"wikipedia:Engineering informatics\" data-key=\"b9306296db1f0d2935eda0c125b63cd9\">advanced engineering informatics<\/a>.\n<\/p><p>Advanced engineering informatics is motivated by the quest to empower engineers to cope with the ever increasing complexity of the systems they have to provide. The discipline strives to provide means that allow engineers to leverage their understanding of the behavior of complex systems through advanced simulation and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">data analysis<\/a> methods. It also strives at improving the collaboration and communication of engineers within the ever more complex collaborative interdisciplinary arrangements they face.\n<\/p><p>Unlike other related disciplines, advanced engineering informatics focuses not on the automation of mundane tasks, but on developing, researching, and exploring methods to enhance the existing work environment of engineers. Advanced engineering informatics scholars believe that well-designed computational methods have the potential to empower engineers in ways that have previously not been possible. They believe that computers cannot only incrementally speed up engineering design work, but significantly disrupt engineering tasks throughout the entire product development life-cycle, from the early stages of conceptual design, to detailed engineering design, to production, to the maintenance of engineered systems.\n<\/p><p>To the above end, advanced engineering informatics acknowledges that engineering work is a <a href=\"https:\/\/www.limswiki.org\/index.php\/Information#As_an_influence_which_leads_to_a_transformation\" title=\"Information\" class=\"wiki-link\" data-key=\"33bf2abbc89b17a3f4ca7bd13bce9e78\">knowledge<\/a>-intensive activity.<sup id=\"rdp-ebb-cite_ref-KunzEdit02_7-0\" class=\"reference\"><a href=\"#cite_note-KunzEdit02-7\">[7]<\/a><\/sup> Any research into how computational methods can support engineering work needs to start with an explicit formalization of the knowledge engineers posses. Advanced engineering informatics is a specific discipline of knowledge engineering<sup id=\"rdp-ebb-cite_ref-SowaPrinc14_8-0\" class=\"reference\"><a href=\"#cite_note-SowaPrinc14-8\">[8]<\/a><\/sup> with an overarching research question: \u201cHow can we formalize complex engineering knowledge to develop advanced computational methods that help engineers to solve practical problems within their constraints and budgets?\u201d\n<\/p><p>With this research question\u2014above and beyond improving our understanding in how to formalize complex engineering knowledge through explicit representations and symbolic or numerical process models\u2014advanced engineering informatics is hence also concerned with understanding how such representations can support practical engineering work. To this end, topics for research are not only the development of advanced computational methods based on explicitly formulated knowledge, but also exploring the representation of information in graphical user interfaces, the provision of extensive knowledge bases through large scale databases, or how engineers and engineering groups can be supported in interpreting solutions and intermediate solution spaces.<sup id=\"rdp-ebb-cite_ref-KunzEdit02_7-1\" class=\"reference\"><a href=\"#cite_note-KunzEdit02-7\">[7]<\/a><\/sup> In all of these endeavors, an explicit focus on engineering knowledge is required to advance this understanding.\n<\/p><p>Despite the scientific and practical importance, most studies published in the scientific engineering journals fail to explicitly address aspects of engineering knowledge formalization and representation. This also holds for publications focusing on the engineering of our built environment. More often than not, new methods, algorithms, or results of data analysis efforts are presented without the contextualization of the suggested methods within a specific engineering context. Often it is not clear how suggested novel methods make use of explicitly formalized engineering knowledge and how the methods support engineers in their knowledge-intensive tasks. By large, the scientific engineering community still needs to establish a continuous growing body of scientific knowledge about how advanced computational methods can support engineers. Consequently, little general understanding about how novel computation methods can be implemented across tasks and engineering disciplines exists. This lack, in turn, has slowed down the development of solutions that could truly enhance practical engineering work.\n<\/p>\n<hr \/>\n<p>This paper is an effort to refocus the current scientific discourse on the importance of engineering knowledge. To this end, we attempt to first provide a clear definition and description of the underlying philosophical basis of knowledge formulation and knowledge engineering as the foundation for all scientific inquiry within the field of advanced engineering informatics. We illustrate these definition and descriptions using a number of recently published articles that focus on the domain of built environment engineering as example.\n<\/p><p>Our second goal for the paper is to start a discussion about the required methodological approaches for advanced engineering informatics <a href=\"https:\/\/www.limswiki.org\/index.php\/Research\" title=\"Research\" class=\"wiki-link\" data-key=\"409634fd90113f119362927fe222f549\">research<\/a> practice. So far there is little to no discourse about research methods within the field, which has significantly hindered its establishment among the other scientific disciplines. To catalyze this discourse, in the second part of the paper we suggest different research approaches and some underlying theories.\n<\/p><p>Of course, like every other scientific discipline, the definitions, concepts, methods, and approaches associated with advanced engineering informatics are an ever-moving target. Therefore, this paper can only represent our current reflections and thinking in the field and is intended to provide food for thought and a catalyst for more reflective and vibrant discussion. By no means are the presented concepts of knowledge formalization and research methods meant as fixed bearing points, but rather as points of departure for wider theoretical explorations. Therefore, the paper also provides an elaborated discussion section with suggestions for future important areas of inquiry.\n<\/p><p>In the next section, we introduce the theoretical underpinnings of knowledge representation and knowledge formalization. That section also illustrates these underpinnings using four recently published research studies. Then different research methods that might be appropriate for advanced engineering informatics research are suggested. Finally, an extensive discussion with suggestions for important research directions are presented, along with conclusions.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Knowledge_representation_and_formalization\">Knowledge representation and formalization<\/span><\/h2>\n<p>Sowa<sup id=\"rdp-ebb-cite_ref-SowaPrinc14_8-1\" class=\"reference\"><a href=\"#cite_note-SowaPrinc14-8\">[8]<\/a><\/sup> defines knowledge engineering as the application of <a href=\"https:\/\/www.limswiki.org\/index.php\/Ontology_(information_science)\" title=\"Ontology (information science)\" class=\"wiki-link\" data-key=\"52d0664bde4b458e81fbc128b911a4a6\">ontology<\/a> and logic to the task of building computational models of some domain for some purpose. To inform advanced engineering informatics research, the definition provided in the introduction is informative as it focuses knowledge engineering on two important aspects. First, the definition suggests building computational models. Hence the definition proposes to move beyond the development of mathematical algorithms, towards models that already make computational prediction about a domain. Second, the definition also implies purpose, therefore requiring a focus on solving practical problems. These two aspects are of utmost importance for all research into advanced engineering informatics. The discipline is not concerned with conceiving new mathematical methods, algorithms, and calculation mechanisms, but rather it is concerned with using such basic computational methods to build models that compute tangible results that are relevant for a specific engineer. Furthermore, this relevance needs to be related to a practical engineering purpose within the wider <a href=\"https:\/\/www.limswiki.org\/index.php\/Systems_development_life_cycle\" title=\"Systems development life cycle\" class=\"wiki-link\" data-key=\"b96939e19621960ee123770c13fa1a84\">product development cycle<\/a> of an engineering system.\n<\/p><p>Moreover, the definition points towards basic methodological approaches that advanced engineering informatics researchers have to be familiar with: ontology, logic, and computation. Within information science, an ontology is a formal representation of all a topic's concepts and their relations. An ontological knowledge representation is concerned with the knowledge of engineers about physical and abstract objects, relations between those objects, and events influencing those objects. Ontological representation allows for a commitment with respect to the model of the specific domain that is required as the basis for any computational method. With this commitment, ontologies help humans and computers understand and fully utilize domain knowledge. One important aspect of advanced engineering informatics research focuses on developing approaches for implementing computer-assisted engineering platforms that apply ontology-based theories and solutions.<sup id=\"rdp-ebb-cite_ref-KotisEnhanc11_9-0\" class=\"reference\"><a href=\"#cite_note-KotisEnhanc11-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HuangDevelop08_10-0\" class=\"reference\"><a href=\"#cite_note-HuangDevelop08-10\">[10]<\/a><\/sup>\n<\/p><p>Each ontology supporting such solutions needs to map the knowledge within a specific universe of discourse.<sup id=\"rdp-ebb-cite_ref-HartmannInform17_11-0\" class=\"reference\"><a href=\"#cite_note-HartmannInform17-11\">[11]<\/a><\/sup> This universe of discourse should be a carefully bounded and focused micro-world<sup id=\"rdp-ebb-cite_ref-SowaPrinc14_8-2\" class=\"reference\"><a href=\"#cite_note-SowaPrinc14-8\">[8]<\/a><\/sup> within an engineering discipline. Alternatively, it could also focus on a specific engineering collaboration between two engineering disciplines. To arrive at computational models as defined above, a bottom-up approach that focuses on a very specific engineering task is required. Moreover, domain ontology schema should be built and updated constantly together with all stakeholders of the knowledge domain. Knowledge is dynamically changing and growing, and, most importantly, it is possessed by multiple domain experts.<sup id=\"rdp-ebb-cite_ref-ValarakosBuild06_12-0\" class=\"reference\"><a href=\"#cite_note-ValarakosBuild06-12\">[12]<\/a><\/sup>\n<\/p><p>The second methodological approach that is suggested by Sowa\u2019s definition is logic. Logic is the systematic study of inference that leads to the acceptance of a specific proposition. Such systematic studies require the clear formalization of a proposition and the development of a set of premises that may or may not support the conclusion. Logic as systematic study allows advanced engineering informatics researchers to formalize rules of inference that engineers use to arrive at conclusions, make decisions, or creatively develop design ideas.\n<\/p><p>In particular, the last point\u2014developing creative design ideas\u2014requires a thorough attention to logic. Currently, more often than not, the formalization of rules of inference can lead to logic that are to rigid or that focus on the formalization of irrelevant inference rules. In these cases, creative engineering design\u2014which is so important for improving complex engineered systems\u2014is inhibited.\n<\/p><p>However, if applied well, logic allows the development of a theory of the intelligent reasoning approaches that engineers follow. Logic allows the formalization of complex engineering understanding about an engineering systems\u2019 behavior across space and time with respect to specific changes of the system under various specific environmental influences. Logic also allows the formalization of knowledge about important procedures that are required during production, or while maintaining an engineering system. Equally important to the formulation of knowledge about processes and procedures is that logic helps engineers account for specific constraints that bound such processes and procedures.\n<\/p><p>Both ontology and knowledge allow engineers to analyze complex engineering knowledge about the structure of an engineering system and its behavior, as well as procedures for its production and maintenance. However, ontology and logic by themselves do not yet allow for the description of engineering purpose. A classical example of this shortcoming is provided by Sowa<sup id=\"rdp-ebb-cite_ref-SowaPrinc14_8-3\" class=\"reference\"><a href=\"#cite_note-SowaPrinc14-8\">[8]<\/a><\/sup>, drawing upon Newton\u2019s second law of motion that relates force, mass, and acceleration. Newton\u2019s equation introduces an ontology that provides a clear and abstract description of the aspects related to the motion of an object. The formula also represents the logic of how force, mass, and acceleration are related. However, the formula itself does not yet propose how an engineer can use it to purposefully analyze a system. An engineer can use the law for three major purposes: to calculate mass from force and acceleration, force from mass and acceleration, or acceleration from mass and force. Which of these purposes is important for an engineer for a specific engineering task can only be formulated by representing the computation that is required within the specific context the engineer is in. Hence, purpose needs to be explicitly formulated while representing and formalizing engineering knowledge.\n<\/p><p>Next to purpose, thought needs to specifically be applied towards the concept of \"context\" while formalizing engineering purpose. To a certain extent, it is impossible to define purpose without such attention to context. At the same time, however, it is important to consider context with respect to the knowledge formalized with ontology and logic. Both ontology and logic are models, and hence it is important to be explicit when and in which circumstances these models are applicable and when these might fail. Hence, understanding context is another important research activity within the field of advanced engineering informatics.\n<\/p><p>It is important for advanced engineering informatics scholars to consider that ontology, logic, and computation can only represent a very abstract model of the reasoning and knowledge of engineers. Formal knowledge representations are by nature fragmented and cannot get close to the true reasoning engineers use to come to their conclusions for specific tasks. No matter how fragmented and abstract ontologies and reasoning are, they, nevertheless, enable efficient communication, not only between engineers but also among advanced engineering informatics scientists.\n<\/p><p>To illustrate the above points, the following sub-sections describe how four recent studies suggested and validated four different computational methods for formalizing complex engineering knowledge within the area of built environment engineering. The examples have been identified as good practice examples by the two authors based on their experience as editors of the journal <i>Advanced Engineering Informatics<\/i>. It was not intended within the scope of this paper to provide a structural literature review, but rather to illustrate the above concepts with a number of loosely selected previous research studies.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Example_1:_Formalizing_engineering_knowledge_with_ontology\">Example 1: Formalizing engineering knowledge with ontology<\/span><\/h3>\n<p>The objective of developing formal ontologies is to help humans and computers understand and, hence, fully utilize domain knowledge in various <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">knowledge management<\/a> systems. Domain ontology schema should be built and updated constantly as a collective intelligence, since knowledge is considered dynamically changing and growing and, most importantly, can be contributed by multiple domain experts.<sup id=\"rdp-ebb-cite_ref-ValarakosBuild06_12-1\" class=\"reference\"><a href=\"#cite_note-ValarakosBuild06-12\">[12]<\/a><\/sup>\n<\/p><p>An example for such a system is Yuan <i>et al.<\/i>\u2018s effort to model the residual value risk around the vulnerability of infrastructure projects.<sup id=\"rdp-ebb-cite_ref-YuanModel18_13-0\" class=\"reference\"><a href=\"#cite_note-YuanModel18-13\">[13]<\/a><\/sup> Financial responsibility on these projects is shared by public and private parties. Understanding financial risks that occur during the delivery life-cycle of such projects is important. Estimating these risks is a complex task that engineers are concerned with already during the conceptual design stages, and that is crucial to thoroughly draft contractual agreements between the public and private partners involved in such projects.\n<\/p><p>Yuan <i>et al.<\/i> formalized the engineering knowledge of this specific domain by proposing an ontology represeting risk sources, risk events, risk consequences, exposures, resilience factors, and contextual sensitivity characteristics that might influence the risks of a specific project. The study also instantiated the ontology formalizing the specific knowledge of an illustrative bridge project and validated the ontology by conducting a survey among domain experts.\n<\/p><p>The study shows the utility of formalizing knowledge using ontologies. The authors illustrate how the ontology allows to visualize the risk factors using knowledge graphs and how these <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_visualization\" title=\"Data visualization\" class=\"wiki-link\" data-key=\"4a3b86cba74bc7bb7471aa3fc2fcccc3\">visualizations<\/a> helped to estimate the financial risks of a project. The study also illustrates how the formal representation of the knowledge allows the computation of automated reasoning paths, for example, to understand the effect of design or environmental changes on a specific risk profile.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Example_2:_Using_logic_to_represent_design_knowledge\">Example 2: Using logic to represent design knowledge<\/span><\/h3>\n<p>An example of how to use logic to formalize engineering knowledge can be found in Min <i>et al.<\/i>\u2018s study that developed rule-based patterns for laying out theme parks.<sup id=\"rdp-ebb-cite_ref-MinARule17_14-0\" class=\"reference\"><a href=\"#cite_note-MinARule17-14\">[14]<\/a><\/sup> Designing leisure spaces in a theme park is a highly knowledge-intensive activity. Theme parks need to provide a highly complex and multi-layered service environment to satisfy visitors. In their study, Min <i>et al.<\/i> identified and formalized patterns used in a number of successful theme parks and combined them into a reasoning system.\n<\/p><p>Some logical patterns formalized in the study are, for example, that facilities such as attractions, restaurants, and shops are equally distributed around a park\u2019s centroid. Another logical pattern Min <i>et al.<\/i> identified and formalized is that building entrances are located at pathways that exhibit relatively low traffic. The authors also illustrated how these patterns can be used by developing a software implementation for theme park design and applying the software to design a new theme park in South Korea. The logic was validated by interviewing experts and by conducting design experiments with four experienced experts.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Example_3:_Optimization\">Example 3: Optimization<\/span><\/h3>\n<p>Much work within the field of advanced engineering informatics has focused on how design optimization can support engineers to identify optimal designs among a set of alternatives. During design optimization, ontology and logic play an important role, as it is required to devise a mathematical formulation of the design problem. To develop this formulation, researchers have to identify variables that describe the alternatives and then relate these variables logically within an objective function that is to be maximized or minimized. Additionally, a number of constraints have to be logically formulated based on the initial design variables. If design problems can be formulated adequately, a large number of computational optimization methods are available that can be applied. While the development of new optimization algorithms would rather fall within the domain of computer science or mathematics, the formulation of design optimization problems is an important topic of advanced engineering informatics research.\n<\/p><p>An example of research that formulates a design optimization problem around a complex engineering task is Jin <i>et al.<\/i>\u2018s study into how to formulate the planning of scaffolding required for complex piping installation work.<sup id=\"rdp-ebb-cite_ref-JinModel17_15-0\" class=\"reference\"><a href=\"#cite_note-JinModel17-15\">[15]<\/a><\/sup> Designing the best set-up for scaffolding is a complex engineering task because of the spatial relationship between the locations in which work needs to be supported and the requirements to set-up supporting structures.\n<\/p><p>For formulating the optimization problem, Jin <i>et al.<\/i> developed a rule-based logic of scaffolding placement and linked these rules with an explicit ontological description combining the timing of construction work tasks, the location of these tasks, and of the pipes\u2019 geometries. The formulation also included different possible postures that workers can be in to install a pipe in an attempt to allow for the optimization of ideal working postures for productive installation work. Different constraints were also formulated such as minimum and maximum acceptable heights for working conditions to bound the automated alternative generation.\n<\/p><p>The formulation of the optimization problem was validated on a practical example concerned with the installation of 71 different pipes for a 21-meter-high industrial plant. This illustrative validation could vividly show how a knowledge representation can help site engineers to set up optimal scaffolding that minimizes the amount of pipes that cannot be installed with a specific set-up while maximizing the productivity of installation work.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Example_4:_Advanced_data_analytics\">Example 4: Advanced data analytics<\/span><\/h3>\n<p>Similar to optimization, the last two decades have seen a large amount of studies that applied advanced data analytics methods, in particular neural network-based machine learning, in an effort to develop methods for supporting engineers. Similar to optimization studies, it is important to focus explicitly on the representation of complex engineering knowledge that is inherent to data analytics studies in order to make those studies meaningful to engineering practice. From a knowledge representation perspective, machine learning methods transform collected data input that engineers can not easily interpret to an output that is interpretable to engineers.<sup id=\"rdp-ebb-cite_ref-SowaPrinc14_8-4\" class=\"reference\"><a href=\"#cite_note-SowaPrinc14-8\">[8]<\/a><\/sup> Advanced engineering informatics research studies these aspects of knowledge representation around the application of well-known data analytics methods from computer science and mathematics. Advanced engineering informatics also explores how such knowledge representations and translations can help engineers to deal with their complex engineering tasks.\n<\/p><p>An example of such a study is the work by Leng <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-LengANew18_16-0\" class=\"reference\"><a href=\"#cite_note-LengANew18-16\">[16]<\/a><\/sup> that developed a forecasting method for wind power systems. In their work, the authors suggest a method to translate wind power signals that are hard to interpret by applying the ridgelet transform method, which allows to mathematically model the singular changes within the wind signal more accurately. Using the wind signals as input features, with the ridgelet transform adjusted, a neural network can be trained to forecast wind power accurately.\n<\/p><p>The authors then demonstrate the utility of the suggested forecasting method by applying it to a wind farm in Alberta. The authors were able to illustrate how the method allows interpretable outputs forecasting wind power for different yearly seasons and even specific days. These predictions can then be used by wind farm engineers to not only design better wind farm layouts, but also to improve the maintenance and management of wind farms in operation.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Research_methods\">Research methods<\/span><\/h2>\n<p>After providing a philosophical basis for research within the field of advanced engineering informatics through focusing the discussion on ontology, logic, purpose, and context, this section will provide some fundamental propositions for how to methodologically approach scientific research. To date, there has been little discussion about research methods in the field, and clear conventions for how to systematically approach research questions have been largely missing. Scientific methods should be empirical in their approach to acquiring knowledge, and they should allow researchers to apply skepticism about what is empirically observed. With this in mind, the section is structured around the two major research tasks of first developing knowledge representations and secondly how to verify, validate, and illustrate them.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Developing_formalizations\">Developing formalizations<\/span><\/h3>\n<p>One of the major research tasks of advanced engineering informatics is the formalization of knowledge with ontology and logic and to represent it. For formalization, well-established methods exist and are widely used. For ontologies, researchers often generate knowledge maps that represent the different identified concepts and that can be created with ontological editors, such as the widely used Prot\u00e9g\u00e9.<sup id=\"rdp-ebb-cite_ref-GennariTheEvo03_17-0\" class=\"reference\"><a href=\"#cite_note-GennariTheEvo03-17\">[17]<\/a><\/sup> Logic is usually represented in mathematical notations, algorithms in pseudo code, or process diagrams. Metamodels, such as the ones listed above, for representing formalizing models of engineering knowledge are in widespread use within the research community.\n<\/p><p>However, little thought has so far been given towards how to systematically elicit complex engineering knowledge from experts and engineering practice. By large, most of the knowledge formalized represents the knowledge of the researchers themselves. Pragmatically, this approach has helped the field to develop because the researchers are often also expert engineers. Scientifically, however, this practice is challenging as it lacks the empirical and systematic character that is asked for by sound scientific practice. Only a few specific methods have been proposed or utilized to elicit engineering knowledge from practitioners.<sup id=\"rdp-ebb-cite_ref-HartmannImplem09_18-0\" class=\"reference\"><a href=\"#cite_note-HartmannImplem09-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SenescuEval13_19-0\" class=\"reference\"><a href=\"#cite_note-SenescuEval13-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JungAuto19_20-0\" class=\"reference\"><a href=\"#cite_note-JungAuto19-20\">[20]<\/a><\/sup> What is common among the few suggested methods is that they either rely upon social science-based methods or on the formal analysis of engineering documents and models.\n<\/p><p>Social science-based methods that have been suggested in the past were based on interviews with engineers and the organization of workshops and focus groups, as well as more experimental methods. One of the more advanced methods proposed is the use of so-called design charrettes. (Senescu and Haymaker, 2013) Design charrettes propose to capture knowledge of engineers by organizing short, intensive design exercises with professionals that can be observed. An analytical analysis of these observations then allow researchers to better understand the knowledge the engineers applied during the charrette for the consecutive structural formalization. As another example of a social science approach, Hartmann <i>et al.<\/i> proposed the ethnographic action research method.<sup id=\"rdp-ebb-cite_ref-HartmannImplem09_18-1\" class=\"reference\"><a href=\"#cite_note-HartmannImplem09-18\">[18]<\/a><\/sup> Ethnographic action research suggests that the researcher gets immersed in engineering work settings by working alongside practicing engineers. They suggest that this immersion allows the researcher to gain the necessary deep insights about practical engineering knowledge prior to formalization.<sup id=\"rdp-ebb-cite_ref-HartmannImplem09_18-2\" class=\"reference\"><a href=\"#cite_note-HartmannImplem09-18\">[18]<\/a><\/sup>\n<\/p><p>Next to social science research methods, researchers have relied on the structured analysis of documents and other artifacts. As discussed earlier, Min <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MinARule17_14-1\" class=\"reference\"><a href=\"#cite_note-MinARule17-14\">[14]<\/a><\/sup>, for example, have formalized design patterns for the design of theme parks. The study identified the patterns through a structured analysis of design documents existing for different realized theme parks.\n<\/p><p>With the advancement of text mining and pattern recognition techniques, recently many studies have been published that use automated methods to extract information from existing design documents and models. This field has developed into a vibrant research area. Automated text mining methods have been applied to formalize engineering knowledge from patents<sup id=\"rdp-ebb-cite_ref-GovindarajanIntell19_21-0\" class=\"reference\"><a href=\"#cite_note-GovindarajanIntell19-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WangANovel19_22-0\" class=\"reference\"><a href=\"#cite_note-WangANovel19-22\">[22]<\/a><\/sup>, identify research trends for building energy savings<sup id=\"rdp-ebb-cite_ref-DingBuild18_23-0\" class=\"reference\"><a href=\"#cite_note-DingBuild18-23\">[23]<\/a><\/sup>, analyze construction site accidents<sup id=\"rdp-ebb-cite_ref-ZhangConstruct19_24-0\" class=\"reference\"><a href=\"#cite_note-ZhangConstruct19-24\">[24]<\/a><\/sup>, predict construction cost overruns<sup id=\"rdp-ebb-cite_ref-WilliamsPredict14_25-0\" class=\"reference\"><a href=\"#cite_note-WilliamsPredict14-25\">[25]<\/a><\/sup>, retrieve CAD drawings<sup id=\"rdp-ebb-cite_ref-YuContent13_26-0\" class=\"reference\"><a href=\"#cite_note-YuContent13-26\">[26]<\/a><\/sup>, or extract best practices from simulation modeling guidelines.<sup id=\"rdp-ebb-cite_ref-KestelOntol19_27-0\" class=\"reference\"><a href=\"#cite_note-KestelOntol19-27\">[27]<\/a><\/sup> Additionally, several graph-based pattern mining methods have been applied to architectural floor plan design<sup id=\"rdp-ebb-cite_ref-StrugReason09_28-0\" class=\"reference\"><a href=\"#cite_note-StrugReason09-28\">[28]<\/a><\/sup> or to the automated extraction and formalization of construction process patterns.<sup id=\"rdp-ebb-cite_ref-SigalovRecog17_29-0\" class=\"reference\"><a href=\"#cite_note-SigalovRecog17-29\">[29]<\/a><\/sup> Finally, geometric pattern recognition techniques have also been developed to support the aerodynamic design of vehicles.<sup id=\"rdp-ebb-cite_ref-GraeningShape14_30-0\" class=\"reference\"><a href=\"#cite_note-GraeningShape14-30\">[30]<\/a><\/sup>\n<\/p><p>Considering that engineers have compiled an extensive digital collection of such design documents, we expect that the years to come will see a further acceleration of this area of research. In practice, the results could provide engineers with dedicated domain-specific search engines that will allow them to better find and understand previous design solutions that are adaptable to a design task at hand. Moreover, the identification of such patterns can lead to an increasing number of evidence-based design tools to support a wide range of engineering tasks. Finally, test mining and pattern matching might allow engineers to derive new insights into the behavior of engineering products and materials, as it will allow the combination of a large amount of previous measurements and test results.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Verification_and_validation\">Verification and validation<\/span><\/h3>\n<p>Next to the question of how to best elicit and formalize knowledge, another question for sound scientific research in the field of advanced engineering informatics is how to ensure that a proposed knowledge formulation is appropriate and useful. Sound scientific practice requires that a knowledge representation is systematically verified and validated. This section will summarize some of the most common approaches used for verification and validation so far.\n<\/p><p>For ontologies, verification is the process of ensuring that the axioms of the ontology reflect the intentions of the author.<sup id=\"rdp-ebb-cite_ref-MatentzogluInfer18_31-0\" class=\"reference\"><a href=\"#cite_note-MatentzogluInfer18-31\">[31]<\/a><\/sup> Building ontologies is an error-prone activity, and it is very difficult to structure ontologies so that they do not allow for unintended inferences, for example, through the introduction of unsatifiable axioms. Ontologies can be verified with the built-in reasoners within common ontology development tools, such as the earlier introduced Prot\u00e9g\u00e9<sup id=\"rdp-ebb-cite_ref-GennariTheEvo03_17-1\" class=\"reference\"><a href=\"#cite_note-GennariTheEvo03-17\">[17]<\/a><\/sup>, which can automatically detect defects in the ontology. While developing ontologies, these automatic reasoners should be used frequently to avoid the propagation of systematic errors early on. As an ontology is growing and evolving, fixing such systematic errors will get increasingly difficult and time-consuming.\n<\/p><p>Verifying a computational method that is based on logic is a more difficult problem. Strictly speaking, sound verification would require mathematically proving the computation. Finding mathematical proofs, however, quickly turns too complex even for relatively simple computations. Another option is a combinatorical approach that controls the outputs of every possible input within the context of the computation. Again, however, true combinatorical efforts are not feasible in real-world research settings, even if the context in which a specific computation should work is carefully bounded. A solution to this problem is the application of sensitivity analysis methods in relation to well defined sampling strategies for different possible combinations of the input values.<sup id=\"rdp-ebb-cite_ref-SaltelliSens00_32-0\" class=\"reference\"><a href=\"#cite_note-SaltelliSens00-32\">[32]<\/a><\/sup> More often than not, however, most studies still simply rely on the use of illustrative examples to verify their suggested computational methods. While such an approach is widely accepted, researchers should at a minimum provide a sound argument for the appropriateness of the used illustrative example in relation to the complexity of the real world engineering challenge at hand.\n<\/p><p>In contrast to verification, validation is the process of evaluating whether the knowledge representation is fit to the engineering purpose at hand. Evaluating the validity requires a closer attention to practical engineering than verification itself and is often more difficult to conduct. While a sound verification only needs to closely evaluate the internal structure of a knowledge representation, validation requires to evaluate a representation within its context and for its specific purpose. Because of this requirement, it is much harder to provide convincing evidence for the validity of a solution, and researchers need to very carefully plan and conduct validation exercises.\n<\/p><p>Validation can be approached at different levels. By far the most often conducted validation and also the most easiest is to apply the knowledge representation to an illustrative example of an engineering task. In this way, it is possible to show that the representation fits its purpose; however, oftentimes such a validation is not very convincing. At a minimum, it is important that the illustrative example used for the validation represents a challenge that engineers would encounter in practice. Far too often, however, studies use radically simplified examples that do not reflect the complexity of a problem that engineers would face in practice. While simplified illustrative examples are a great vehicle for verification, such efforts can hardly be considered actual validation.\n<\/p><p>Other approaches that can be used to validate knowledge representation systems center around the implementation of a prototype of a computational method that implements the representation. Simple efforts use such a prototype to establish a demonstration of the system at work and then ask engineers for their opinion on the system. Again, more often than not, such efforts to validation are not very convincing. The methodological problem with the approach is that it is hard to develop a good sampling strategy to target engineers that reflect different levels of expertise and backgrounds that would be required to be able to generalize the results. Moreover, even if a sound sampling strategy can be developed it is very hard to ensure participation of engineers in such studies. Some researchers, therefore, have reverted to student surveys, which are even less convincing.\n<\/p><p>A slightly more advanced approach to validation is to ask engineers or engineering students to use the prototype for solving an engineering task. Such approaches can provide much stronger evidence, but any prototype also needs to be designed very carefully. For one, there often is the tendency to over-structure the experimental setting, giving the test subjects an overly targeted assignment for using the prototype. Moreover, some studies have then compared such efforts with a control group of engineers that have solved the same task without the support of the prototype. While seemingly providing strong validation for the working of the prototype, such efforts can hardly show the utility of the knowledge representation to empower engineers to cope with an engineering problem in a significantly different manner than was possible before.\n<\/p><p>A less structured effort to validation that centers around devising open-ended experimental settings in which engineers solve a design challenge without too much structuring of the process often proves to be more convincing. Such open-ended design experiments resemble the earlier introduced design charette studies, albeit this time the designers use a prototypical implementation for the design task at hand. While it is possible to ask participants in these studies questions before and after the experiment, the experiment itself should be closely observed and ideally recorded. The observations and recordings can then be analyzed using qualitative research methods that are well established in the social sciences.<sup id=\"rdp-ebb-cite_ref-MilesQual94_33-0\" class=\"reference\"><a href=\"#cite_note-MilesQual94-33\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CorbinBasics15_34-0\" class=\"reference\"><a href=\"#cite_note-CorbinBasics15-34\">[34]<\/a><\/sup> Such qualitative data analysis methods can often provide very strong evidence that the prototype empowered engineers in the design experiments to work in a substantially different manner than in current practice. Again, such experiments can be conducted with students as well, but, of course, it is more convincing if practicing engineers can be convinced to participate.\n<\/p><p>By far the most convincing validation is to show that a prototype or even an already more developed system design based upon a structured knowledge representation can support practicing engineers. Unfortunately, collecting such evidence is seldom feasible as engineers need to be found that are willing and able to use a new tool in practice. The ethnographic action research method<sup id=\"rdp-ebb-cite_ref-HartmannImplem09_18-3\" class=\"reference\"><a href=\"#cite_note-HartmannImplem09-18\">[18]<\/a><\/sup> can serve as a starting point for slowly convincing and training practitioners by working with them for a prolonged period. While engineers work on a task, the researcher can shadow the work of the engineers by implementing observed decision making and design activities using a prototypical system. In this manner, evidence for the utility of the prototype can be built up step-by-step, and engineers can be convinced little by little to implement the system directly. Complicating the situation for the researcher, all these efforts need to be closely monitored by structurally collecting data and documenting evidence. Again qualitative data collection and analysis methods from the social sciences<sup id=\"rdp-ebb-cite_ref-MilesQual94_33-1\" class=\"reference\"><a href=\"#cite_note-MilesQual94-33\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CorbinBasics15_34-1\" class=\"reference\"><a href=\"#cite_note-CorbinBasics15-34\">[34]<\/a><\/sup> can help with such endeavors.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>This paper has developed a philosophical foundation for establishing a more coherent field of advanced engineering informatics that is solidly based upon formalizing complex engineering knowledge. This foundation suggests that ontology and logic form the basis for scientific knowledge formalization efforts in the field, and associated research needs to be concerned with purpose and context. The paper also suggests a number of research approaches that can help to elicit knowledge from engineers for formalization and help researchers to design studies for the verification and validation of such formulations. All of the above discussions and examples do have a focus on built environment engineering, a discipline that provides some of the most complex engineering systems to date.\n<\/p><p>The suggested approaches are cumbersome, particularly since engineers in practice posses an enormous amount of background knowledge, have multiple often conflicting purposes, and work in a myriad of differing, complex settings. Even worse, on top of this highly heterogeneous character of knowledge, that knowledge is also fluid and ever-changing.\n<\/p><p>To cope with these difficulties, we suggest that scientific research in the field is approached bottom-up, paying detailed attention to the specific context and purpose of engineers. Knowledge can only be explored in small chunks that correspond to very specific engineering purposes and contexts. Advanced engineering informatics must therefore be an ever-evolving research field that is advanced in small steps. In turn, generalizations, definitions, and anticipations will always be inconsistent. Conditions observed within a specific context might be abnormal for other contexts, developed solutions for specific purposes might be conflicting in other contexts, and applications might have unanticipated outcomes that can only be recognized much later.\n<\/p><p>In all, it is unlikely that scientists working in the field of advanced engineering informatics will discover great breakthroughs. Advanced engineering informatics is a humble research discipline that carefully needs to be built up slowly with a growing body of knowledge that is continuously challenged, criticized and revised. Studies need to be designed that carefully build upon each other. Each single research project, PhD thesis, or scientific publication can only contribute a little chunk of knowledge to the advancement of the field.\n<\/p><p>With this in mind, the field also needs to carefully review its current practice in the sense of whether studies rigorously and explicitly built upon previous work. It is often far too easy to start a study from scratch, develop a new computational method, and claim its utility by illustrating its use on some self-proclaimed engineering problem that is to be solved. Such studies, however, lack the required scientific rigor in providing empirical evidence and do little to move the field systematically forward. Moreover, such studies often fail to empower engineers with methods to cope with the ever increasing complexity of engineering systems they need to design, produce, and maintain.\n<\/p><p>To provide true scientific stepping stones that advance our knowledge, researchers need to thoroughly understand the body of research that has been conducted previously. Literature reviews that inform studies should both carefully summarize work that has been done to support similar engineering purposes, for example, by thoroughly understanding engineering disciplines, tasks, and contexts that have been explored in previous studies. At the same time, researchers have to develop a deep understanding about how the specific set of computational methods that they intend to apply for formalizing knowledge has been used to solve engineering problems in a wide range of other contexts. Literature reviews for advanced engineering informatics studies, therefore, always need to be twofold evaluating the state of the art in supporting a specific engineering context and evaluating the state of the art of a specific computational method. Only then will a consistent academic stream of inquiry across time and space emerge that allows to slowly generalize findings to answer the two main research questions of the field: \u201cHow do we best formalize complex engineering knowledge with novel computational methods?\u201d and \u201cHow can engineers be empowered by computational methods to significantly improve their work?\u201d\n<\/p><p>Another important aspect is that reproductive studies that further verify and validate previous studies are important to advance knowledge. Again, far too often researchers choose not to conduct reproductive studies or truthfully frame their conducted work as reproductive. The lure of being able to claim a significant scientific breakthrough is often too tempting. In the defense of the scientists, far too often, reviewers of scientific work also fail to acknowledge the important aspect of reproductive studies. To become a more coherent field, it is important that attitudes change. Studies that can replicate earlier findings, even in close and similar engineering contexts, should be considered as important. These studies can increase the sophistication of validation methods, provide supporting or debunking evidence, and further discuss insights using additional empirical evidence collected. The field should consider such studies as equally important for advancing our knowledge than studies that unconvincingly claim the utility of newly conceived methods.\n<\/p><p>Despite all these problems and the cumbersome suggested research that is required to formalize engineering knowledge, the clear communication that formal knowledge representation allows for will allow others to continuously improve the utility of a specific computational approach in empowering engineers in ways that have previously not been possible. Additionally, clear communication will allow for the development of shared models for achieving the integration of different engineering disciplines, which are required to design, produce, and maintain our ever more complex engineered systems. Finally, such an explicit focus on knowledge representation with ontology and logic, together with a specific focus on purpose and context, will allow for the generalized application of different methods across engineering fields and hence further establish the scientific discipline of advanced engineering informatics.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>To refocus current research efforts in developing computational methods within the wider field of engineering, and in particular with respect to built environment engineering, we argue that knowledge representation is the main research effort that is required to develop technologies that not only automate mundane engineering tasks, but also provide engineers with tools that will allow them to do things they were not able to do before. We suggest that such tools will be required so that engineers can deal with the ever increasing complexity of the modern engineering systems they need to deliver.\n<\/p><p>To focus scientific work in the field of advanced engineering informatics on knowledge representation, we first introduce the underlying philosophical concepts of knowledge representation and formalization. To this end, we heavily draw on the seminal work of Sowa.<sup id=\"rdp-ebb-cite_ref-SowaPrinc14_8-5\" class=\"reference\"><a href=\"#cite_note-SowaPrinc14-8\">[8]<\/a><\/sup> We then illustrate these concepts using four recently published studies. Based on the theoretical concepts, we share our thoughts about possible research methods that scholars can draw upon while developing and empirically validating knowledge representations. The suggested research methods are meant to start an ongoing discussion about how to best conduct research in the field of advanced engineering informatics.\n<\/p><p>In conclusion, we hope that this position paper can help scientists to understand the field of advanced engineering informatics and its importance better. We also hope that the paper can support scholars in designing studies within the field that can improve our knowledge of how best to use computational methods to formalize complex engineering knowledge. As a follow-up step to this paper, we suggest that researchers conduct a structured and comprehensive literature review to further expound upon the field of advanced engineering informatics.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-MitropoulosForces00-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MitropoulosForces00_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mitropoulos, P.; Tatum, C.B. (2000). \"Forces Driving Adoption of New Information Technologies\". <i>Journal of Construction Engineering and Management<\/i> <b>126<\/b> (5): 340\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1061%2F%28ASCE%290733-9364%282000%29126%3A5%28340%29\" target=\"_blank\">10.1061\/(ASCE)0733-9364(2000)126:5(340)<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Forces+Driving+Adoption+of+New+Information+Technologies&rft.jtitle=Journal+of+Construction+Engineering+and+Management&rft.aulast=Mitropoulos%2C+P.%3B+Tatum%2C+C.B.&rft.au=Mitropoulos%2C+P.%3B+Tatum%2C+C.B.&rft.date=2000&rft.volume=126&rft.issue=5&rft.pages=340%E2%80%938&rft_id=info:doi\/10.1061%2F%28ASCE%290733-9364%282000%29126%3A5%28340%29&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LinderothUnder10-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LinderothUnder10_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Linderoth, H.C.J. (2010). \"Understanding adoption and use of BIM as the creation of actor networks\". <i>Automation in Construction<\/i> <b>19<\/b> (1): 66\u201372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.autcon.2009.09.003\" target=\"_blank\">10.1016\/j.autcon.2009.09.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+adoption+and+use+of+BIM+as+the+creation+of+actor+networks&rft.jtitle=Automation+in+Construction&rft.aulast=Linderoth%2C+H.C.J.&rft.au=Linderoth%2C+H.C.J.&rft.date=2010&rft.volume=19&rft.issue=1&rft.pages=66%E2%80%9372&rft_id=info:doi\/10.1016%2Fj.autcon.2009.09.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DavisResist09-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DavisResist09_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Davis, K.A.; Songer, A.D. (2009). \"Resistance to IT Change in the AEC Industry: Are the Stereotypes True?\". <i>Journal of Construction Engineering and Management<\/i> <b>135<\/b> (12): 1324-1333. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1061%2F%28ASCE%29CO.1943-7862.0000108\" target=\"_blank\">10.1061\/(ASCE)CO.1943-7862.0000108<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Resistance+to+IT+Change+in+the+AEC+Industry%3A+Are+the+Stereotypes+True%3F&rft.jtitle=Journal+of+Construction+Engineering+and+Management&rft.aulast=Davis%2C+K.A.%3B+Songer%2C+A.D.&rft.au=Davis%2C+K.A.%3B+Songer%2C+A.D.&rft.date=2009&rft.volume=135&rft.issue=12&rft.pages=1324-1333&rft_id=info:doi\/10.1061%2F%28ASCE%29CO.1943-7862.0000108&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeWeckEngineering11-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeWeckEngineering11_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">de Weck, O.L.; Roos, D.; Magee, C.L. (2011). <i>Engineering Systems: Meeting Human Needs in a Complex Technological World<\/i>. MIT Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780262016704.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Engineering+Systems%3A+Meeting+Human+Needs+in+a+Complex+Technological+World&rft.aulast=de+Weck%2C+O.L.%3B+Roos%2C+D.%3B+Magee%2C+C.L.&rft.au=de+Weck%2C+O.L.%3B+Roos%2C+D.%3B+Magee%2C+C.L.&rft.date=2011&rft.pub=MIT+Press&rft.isbn=9780262016704&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HartmannAlign12-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HartmannAlign12_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hartmann, T.; van Meerveld, H.; Vossebeld, N. et al. (2012). \"Aligning building information model tools and construction management methods\". <i>Automation in Construction<\/i> <b>22<\/b> (1): 605\u201313. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.autcon.2011.12.011\" target=\"_blank\">10.1016\/j.autcon.2011.12.011<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Aligning+building+information+model+tools+and+construction+management+methods&rft.jtitle=Automation+in+Construction&rft.aulast=Hartmann%2C+T.%3B+van+Meerveld%2C+H.%3B+Vossebeld%2C+N.+et+al.&rft.au=Hartmann%2C+T.%3B+van+Meerveld%2C+H.%3B+Vossebeld%2C+N.+et+al.&rft.date=2012&rft.volume=22&rft.issue=1&rft.pages=605%E2%80%9313&rft_id=info:doi\/10.1016%2Fj.autcon.2011.12.011&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HartmannGoal11-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HartmannGoal11_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hartmann, T. (2011). \"Goal and Process Alignment during the Implementation of Decision Support Systems by Project Teams\". <i>Journal of Construction Engineering and Management<\/i> <b>137<\/b> (12): 1134\u201341. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1061%2F%28ASCE%29CO.1943-7862.0000389\" target=\"_blank\">10.1061\/(ASCE)CO.1943-7862.0000389<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Goal+and+Process+Alignment+during+the+Implementation+of+Decision+Support+Systems+by+Project+Teams&rft.jtitle=Journal+of+Construction+Engineering+and+Management&rft.aulast=Hartmann%2C+T.&rft.au=Hartmann%2C+T.&rft.date=2011&rft.volume=137&rft.issue=12&rft.pages=1134%E2%80%9341&rft_id=info:doi\/10.1061%2F%28ASCE%29CO.1943-7862.0000389&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KunzEdit02-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KunzEdit02_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-KunzEdit02_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kunz, J.C.; Smith, I.F.C.; Tomiyama, T. (2002). \"Editorial\". <i>Advanced Engineering Informatics<\/i> <b>16<\/b> (1): 1\u20132. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS1474-0346%2802%2900004-6\" target=\"_blank\">10.1016\/S1474-0346(02)00004-6<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Editorial&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Kunz%2C+J.C.%3B+Smith%2C+I.F.C.%3B+Tomiyama%2C+T.&rft.au=Kunz%2C+J.C.%3B+Smith%2C+I.F.C.%3B+Tomiyama%2C+T.&rft.date=2002&rft.volume=16&rft.issue=1&rft.pages=1%E2%80%932&rft_id=info:doi\/10.1016%2FS1474-0346%2802%2900004-6&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SowaPrinc14-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SowaPrinc14_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-SowaPrinc14_8-1\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-SowaPrinc14_8-2\">8.2<\/a><\/sup> <sup><a href=\"#cite_ref-SowaPrinc14_8-3\">8.3<\/a><\/sup> <sup><a href=\"#cite_ref-SowaPrinc14_8-4\">8.4<\/a><\/sup> <sup><a href=\"#cite_ref-SowaPrinc14_8-5\">8.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Sowa, J.F. (2014). <i>Principles of Semantic Networks: Explorations in the Representation of Knowledge<\/i>. Elsevier. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781483221144.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Principles+of+Semantic+Networks%3A+Explorations+in+the+Representation+of+Knowledge&rft.aulast=Sowa%2C+J.F.&rft.au=Sowa%2C+J.F.&rft.date=2014&rft.pub=Elsevier&rft.isbn=9781483221144&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KotisEnhanc11-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KotisEnhanc11_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kotis, K.; Papasalourou, A.; Vouros, G. et al. (2011). \"Enhancing the Collective Knowledge for the Engineering of Ontologies in Open and Socially Constructed Learning Spaces\". <i>Journal of Universal Computer Science<\/i> <b>17<\/b> (12): 1710\u201342. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3217%2Fjucs-017-12-1710\" target=\"_blank\">10.3217\/jucs-017-12-1710<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enhancing+the+Collective+Knowledge+for+the+Engineering+of+Ontologies+in+Open+and+Socially+Constructed+Learning+Spaces&rft.jtitle=Journal+of+Universal+Computer+Science&rft.aulast=Kotis%2C+K.%3B+Papasalourou%2C+A.%3B+Vouros%2C+G.+et+al.&rft.au=Kotis%2C+K.%3B+Papasalourou%2C+A.%3B+Vouros%2C+G.+et+al.&rft.date=2011&rft.volume=17&rft.issue=12&rft.pages=1710%E2%80%9342&rft_id=info:doi\/10.3217%2Fjucs-017-12-1710&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuangDevelop08-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HuangDevelop08_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Huang, C.J.; Trappey, A.J.C.; Wu, C.Y. (2008). \"Develop a Formal Ontology Engineering Methodology for Technical Knowledge Definition in R&D Knowledge Management\". In Curran, R.; Chou, S.Y.; Trappey, A.. <i>Collaborative Product and Service Life Cycle Management for a Sustainable World<\/i>. Springer. pp. 495\u2013502. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-1-84800-972-1_46\" target=\"_blank\">10.1007\/978-1-84800-972-1_46<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781848009721.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Develop+a+Formal+Ontology+Engineering+Methodology+for+Technical+Knowledge+Definition+in+R%26D+Knowledge+Management&rft.atitle=Collaborative+Product+and+Service+Life+Cycle+Management+for+a+Sustainable+World&rft.aulast=Huang%2C+C.J.%3B+Trappey%2C+A.J.C.%3B+Wu%2C+C.Y.&rft.au=Huang%2C+C.J.%3B+Trappey%2C+A.J.C.%3B+Wu%2C+C.Y.&rft.date=2008&rft.pages=pp.%26nbsp%3B495%E2%80%93502&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-1-84800-972-1_46&rft.isbn=9781848009721&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HartmannInform17-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HartmannInform17_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hartmann, T.; Amor, R.; East, E.W. (2017). \"Information Model Purposes in Building and Facility Design\". <i>Journal of Computing in Civil Engineering<\/i> <b>31<\/b> (6): 04017054. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1061%2F%28ASCE%29CP.1943-5487.0000706\" target=\"_blank\">10.1061\/(ASCE)CP.1943-5487.0000706<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+Model+Purposes+in+Building+and+Facility+Design&rft.jtitle=Journal+of+Computing+in+Civil+Engineering&rft.aulast=Hartmann%2C+T.%3B+Amor%2C+R.%3B+East%2C+E.W.&rft.au=Hartmann%2C+T.%3B+Amor%2C+R.%3B+East%2C+E.W.&rft.date=2017&rft.volume=31&rft.issue=6&rft.pages=04017054&rft_id=info:doi\/10.1061%2F%28ASCE%29CP.1943-5487.0000706&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ValarakosBuild06-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ValarakosBuild06_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-ValarakosBuild06_12-1\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Valarakos, A.G.; Karkaletsis, V.; Alexopoulou, D. et al. (2006). \"Building an allergens ontology and maintaining it using machine learning techniques\". <i>Computers in Biology and Medicine<\/i> <b>36<\/b> (10): 1155-1184. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.compbiomed.2005.09.007\" target=\"_blank\">10.1016\/j.compbiomed.2005.09.007<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+an+allergens+ontology+and+maintaining+it+using+machine+learning+techniques&rft.jtitle=Computers+in+Biology+and+Medicine&rft.aulast=Valarakos%2C+A.G.%3B+Karkaletsis%2C+V.%3B+Alexopoulou%2C+D.+et+al.&rft.au=Valarakos%2C+A.G.%3B+Karkaletsis%2C+V.%3B+Alexopoulou%2C+D.+et+al.&rft.date=2006&rft.volume=36&rft.issue=10&rft.pages=1155-1184&rft_id=info:doi\/10.1016%2Fj.compbiomed.2005.09.007&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YuanModel18-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YuanModel18_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yuan, J.; Li, X.; Chen, K. et al. (2018). \"Modelling residual value risk through ontology to address vulnerability of PPP project system\". <i>Advanced Engineering Informatics<\/i> <b>38<\/b>: 776-793. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2018.10.009\" target=\"_blank\">10.1016\/j.aei.2018.10.009<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Modelling+residual+value+risk+through+ontology+to+address+vulnerability+of+PPP+project+system&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Yuan%2C+J.%3B+Li%2C+X.%3B+Chen%2C+K.+et+al.&rft.au=Yuan%2C+J.%3B+Li%2C+X.%3B+Chen%2C+K.+et+al.&rft.date=2018&rft.volume=38&rft.pages=776-793&rft_id=info:doi\/10.1016%2Fj.aei.2018.10.009&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MinARule17-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MinARule17_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-MinARule17_14-1\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Min, D.A.; Hyun, K.H.; Kim, S.-J. et al. (2017). \"A rule-based servicescape design support system from the design patterns of theme parks\". <i>Advanced Engineering Informatics<\/i> <b>32<\/b>: 77\u201391. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2017.01.005\" target=\"_blank\">10.1016\/j.aei.2017.01.005<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+rule-based+servicescape+design+support+system+from+the+design+patterns+of+theme+parks&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Min%2C+D.A.%3B+Hyun%2C+K.H.%3B+Kim%2C+S.-J.+et+al.&rft.au=Min%2C+D.A.%3B+Hyun%2C+K.H.%3B+Kim%2C+S.-J.+et+al.&rft.date=2017&rft.volume=32&rft.pages=77%E2%80%9391&rft_id=info:doi\/10.1016%2Fj.aei.2017.01.005&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JinModel17-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JinModel17_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jin, H.; Nahangi, M.; Goodrum, P.M. et al. (2017). \"Model-based space planning for temporary structures using simulation-based multi-objective programming\". <i>Advanced Engineering Informatics<\/i> <b>33<\/b>: 164\u201380. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2017.07.001\" target=\"_blank\">10.1016\/j.aei.2017.07.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Model-based+space+planning+for+temporary+structures+using+simulation-based+multi-objective+programming&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Jin%2C+H.%3B+Nahangi%2C+M.%3B+Goodrum%2C+P.M.+et+al.&rft.au=Jin%2C+H.%3B+Nahangi%2C+M.%3B+Goodrum%2C+P.M.+et+al.&rft.date=2017&rft.volume=33&rft.pages=164%E2%80%9380&rft_id=info:doi\/10.1016%2Fj.aei.2017.07.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LengANew18-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LengANew18_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leng, H.; Li, X.; Zhu, J. et al. (2018). \"A new wind power prediction method based on ridgelet transforms, hybrid feature selection and closed-loop forecasting\". <i>Advanced Engineering Informatics<\/i> <b>36<\/b>: 20\u201330. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2018.02.006\" target=\"_blank\">10.1016\/j.aei.2018.02.006<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+new+wind+power+prediction+method+based+on+ridgelet+transforms%2C+hybrid+feature+selection+and+closed-loop+forecasting&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Leng%2C+H.%3B+Li%2C+X.%3B+Zhu%2C+J.+et+al.&rft.au=Leng%2C+H.%3B+Li%2C+X.%3B+Zhu%2C+J.+et+al.&rft.date=2018&rft.volume=36&rft.pages=20%E2%80%9330&rft_id=info:doi\/10.1016%2Fj.aei.2018.02.006&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GennariTheEvo03-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GennariTheEvo03_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-GennariTheEvo03_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gennari, J.H.; Musen, M.A.; Fergerson, R.W. et al. (2003). \"The evolution of Prot\u00e9g\u00e9: An environment for knowledge-based systems development\". <i>International Journal of Human-Computer Studies<\/i> <b>58<\/b> (1): 89\u2013123. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS1071-5819%2802%2900127-1\" target=\"_blank\">10.1016\/S1071-5819(02)00127-1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+evolution+of+Prot%C3%A9g%C3%A9%3A+An+environment+for+knowledge-based+systems+development&rft.jtitle=International+Journal+of+Human-Computer+Studies&rft.aulast=Gennari%2C+J.H.%3B+Musen%2C+M.A.%3B+Fergerson%2C+R.W.+et+al.&rft.au=Gennari%2C+J.H.%3B+Musen%2C+M.A.%3B+Fergerson%2C+R.W.+et+al.&rft.date=2003&rft.volume=58&rft.issue=1&rft.pages=89%E2%80%93123&rft_id=info:doi\/10.1016%2FS1071-5819%2802%2900127-1&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HartmannImplem09-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HartmannImplem09_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-HartmannImplem09_18-1\">18.1<\/a><\/sup> <sup><a href=\"#cite_ref-HartmannImplem09_18-2\">18.2<\/a><\/sup> <sup><a href=\"#cite_ref-HartmannImplem09_18-3\">18.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hartmann, T.; Fischer, M.; Haymaker, J. (2009). \"Implementing information systems with project teams using ethnographic\u2013action research\". <i>Advanced Engineering Informatics<\/i> <b>23<\/b> (1): 57\u201367. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2008.06.006\" target=\"_blank\">10.1016\/j.aei.2008.06.006<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementing+information+systems+with+project+teams+using+ethnographic%E2%80%93action+research&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Hartmann%2C+T.%3B+Fischer%2C+M.%3B+Haymaker%2C+J.&rft.au=Hartmann%2C+T.%3B+Fischer%2C+M.%3B+Haymaker%2C+J.&rft.date=2009&rft.volume=23&rft.issue=1&rft.pages=57%E2%80%9367&rft_id=info:doi\/10.1016%2Fj.aei.2008.06.006&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SenescuEval13-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SenescuEval13_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Senescu, R.R.; Haymaker, J.R. (2013). \"Evaluating and improving the effectiveness and efficiency of design process communication\". <i>Advanced Engineering Informatics<\/i> <b>27<\/b> (2): 299\u2013313. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2013.01.003\" target=\"_blank\">10.1016\/j.aei.2013.01.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluating+and+improving+the+effectiveness+and+efficiency+of+design+process+communication&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Senescu%2C+R.R.%3B+Haymaker%2C+J.R.&rft.au=Senescu%2C+R.R.%3B+Haymaker%2C+J.R.&rft.date=2013&rft.volume=27&rft.issue=2&rft.pages=299%E2%80%93313&rft_id=info:doi\/10.1016%2Fj.aei.2013.01.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JungAuto19-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JungAuto19_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jung, N.; Lee, G. (2019). \"Automated classification of building information modeling (BIM) case studies by BIM use based on natural language processing (NLP) and unsupervised learning\". <i>Advanced Engineering Informatics<\/i> <b>41<\/b>: 100917. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2019.04.007\" target=\"_blank\">10.1016\/j.aei.2019.04.007<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automated+classification+of+building+information+modeling+%28BIM%29+case+studies+by+BIM+use+based+on+natural+language+processing+%28NLP%29+and+unsupervised+learning&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Jung%2C+N.%3B+Lee%2C+G.&rft.au=Jung%2C+N.%3B+Lee%2C+G.&rft.date=2019&rft.volume=41&rft.pages=100917&rft_id=info:doi\/10.1016%2Fj.aei.2019.04.007&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GovindarajanIntell19-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GovindarajanIntell19_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Govindarajan, U.H.; Trappey, A.J.C.; Trappey, C.V. (2019). \"Intelligent collaborative patent mining using excessive topic generation\". <i>Advanced Engineering Informatics<\/i> <b>42<\/b>: 100955. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2019.100955\" target=\"_blank\">10.1016\/j.aei.2019.100955<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Intelligent+collaborative+patent+mining+using+excessive+topic+generation&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Govindarajan%2C+U.H.%3B+Trappey%2C+A.J.C.%3B+Trappey%2C+C.V.&rft.au=Govindarajan%2C+U.H.%3B+Trappey%2C+A.J.C.%3B+Trappey%2C+C.V.&rft.date=2019&rft.volume=42&rft.pages=100955&rft_id=info:doi\/10.1016%2Fj.aei.2019.100955&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangANovel19-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangANovel19_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, J.; Chen, Y.-J. (2019). \"A novelty detection patent mining approach for analyzing technological opportunities\". <i>Advanced Engineering Informatics<\/i> <b>42<\/b>: 100941. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2019.100941\" target=\"_blank\">10.1016\/j.aei.2019.100941<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+novelty+detection+patent+mining+approach+for+analyzing+technological+opportunities&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Wang%2C+J.%3B+Chen%2C+Y.-J.&rft.au=Wang%2C+J.%3B+Chen%2C+Y.-J.&rft.date=2019&rft.volume=42&rft.pages=100941&rft_id=info:doi\/10.1016%2Fj.aei.2019.100941&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DingBuild18-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DingBuild18_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ding, Z.; Li, Z.; Fan, C. (2018). \"Building energy savings: Analysis of research trends based on text mining\". <i>Automation in Construction<\/i> <b>96<\/b>: 398\u2013410. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.autcon.2018.10.008\" target=\"_blank\">10.1016\/j.autcon.2018.10.008<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+energy+savings%3A+Analysis+of+research+trends+based+on+text+mining&rft.jtitle=Automation+in+Construction&rft.aulast=Ding%2C+Z.%3B+Li%2C+Z.%3B+Fan%2C+C.&rft.au=Ding%2C+Z.%3B+Li%2C+Z.%3B+Fan%2C+C.&rft.date=2018&rft.volume=96&rft.pages=398%E2%80%93410&rft_id=info:doi\/10.1016%2Fj.autcon.2018.10.008&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhangConstruct19-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhangConstruct19_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhang, F.; Fleyeh, H.; Wang, X. et al. (2019). \"Construction site accident analysis using text mining and natural language processing techniques\". <i>Automation in Construction<\/i> <b>99<\/b>: 238\u201348. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.autcon.2018.12.016\" target=\"_blank\">10.1016\/j.autcon.2018.12.016<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Construction+site+accident+analysis+using+text+mining+and+natural+language+processing+techniques&rft.jtitle=Automation+in+Construction&rft.aulast=Zhang%2C+F.%3B+Fleyeh%2C+H.%3B+Wang%2C+X.+et+al.&rft.au=Zhang%2C+F.%3B+Fleyeh%2C+H.%3B+Wang%2C+X.+et+al.&rft.date=2019&rft.volume=99&rft.pages=238%E2%80%9348&rft_id=info:doi\/10.1016%2Fj.autcon.2018.12.016&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilliamsPredict14-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilliamsPredict14_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Williams, T.P.; Gong, J. (2014). \"Predicting construction cost overruns using text mining, numerical data and ensemble classifiers\". <i>Automation in Construction<\/i> <b>43<\/b>: 23\u201329. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.autcon.2014.02.014\" target=\"_blank\">10.1016\/j.autcon.2014.02.014<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Predicting+construction+cost+overruns+using+text+mining%2C+numerical+data+and+ensemble+classifiers&rft.jtitle=Automation+in+Construction&rft.aulast=Williams%2C+T.P.%3B+Gong%2C+J.&rft.au=Williams%2C+T.P.%3B+Gong%2C+J.&rft.date=2014&rft.volume=43&rft.pages=23%E2%80%9329&rft_id=info:doi\/10.1016%2Fj.autcon.2014.02.014&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YuContent13-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YuContent13_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yu, W.-D.; Hsu, J.-Y. (2013). \"Content-based text mining technique for retrieval of CAD documents\". <i>Automation in Construction<\/i> <b>31<\/b>: 65\u201374. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.autcon.2012.11.037\" target=\"_blank\">10.1016\/j.autcon.2012.11.037<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Content-based+text+mining+technique+for+retrieval+of+CAD+documents&rft.jtitle=Automation+in+Construction&rft.aulast=Yu%2C+W.-D.%3B+Hsu%2C+J.-Y.&rft.au=Yu%2C+W.-D.%3B+Hsu%2C+J.-Y.&rft.date=2013&rft.volume=31&rft.pages=65%E2%80%9374&rft_id=info:doi\/10.1016%2Fj.autcon.2012.11.037&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KestelOntol19-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KestelOntol19_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kestel, P.; K\u00fcgler, P.; Zirngibl, C. et al. (2019). \"Ontology-based approach for the provision of simulation knowledge acquired by Data and Text Mining processes\". <i>Advanced Engineering Informatics<\/i> <b>39<\/b>: 292\u2013305. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2019.02.001\" target=\"_blank\">10.1016\/j.aei.2019.02.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ontology-based+approach+for+the+provision+of+simulation+knowledge+acquired+by+Data+and+Text+Mining+processes&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Kestel%2C+P.%3B+K%C3%BCgler%2C+P.%3B+Zirngibl%2C+C.+et+al.&rft.au=Kestel%2C+P.%3B+K%C3%BCgler%2C+P.%3B+Zirngibl%2C+C.+et+al.&rft.date=2019&rft.volume=39&rft.pages=292%E2%80%93305&rft_id=info:doi\/10.1016%2Fj.aei.2019.02.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StrugReason09-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StrugReason09_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Strug, B.; \u015alusarczyk, G. (2009). \"Reasoning about designs through frequent patterns mining\". <i>Advanced Engineering Informatics<\/i> <b>23<\/b> (4): 361\u201369. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2009.06.007\" target=\"_blank\">10.1016\/j.aei.2009.06.007<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reasoning+about+designs+through+frequent+patterns+mining&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Strug%2C+B.%3B+%C5%9Alusarczyk%2C+G.&rft.au=Strug%2C+B.%3B+%C5%9Alusarczyk%2C+G.&rft.date=2009&rft.volume=23&rft.issue=4&rft.pages=361%E2%80%9369&rft_id=info:doi\/10.1016%2Fj.aei.2009.06.007&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SigalovRecog17-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SigalovRecog17_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sigalov, K.; K\u00f6nig, M. (2017). \"Recognition of process patterns for BIM-based construction schedules\". <i>Advanced Engineering Informatics<\/i> <b>33<\/b>: 456\u201372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2016.12.003\" target=\"_blank\">10.1016\/j.aei.2016.12.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Recognition+of+process+patterns+for+BIM-based+construction+schedules&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Sigalov%2C+K.%3B+K%C3%B6nig%2C+M.&rft.au=Sigalov%2C+K.%3B+K%C3%B6nig%2C+M.&rft.date=2017&rft.volume=33&rft.pages=456%E2%80%9372&rft_id=info:doi\/10.1016%2Fj.aei.2016.12.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GraeningShape14-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GraeningShape14_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Graening, L.; Sendhoff, B. (2014). \"Shape mining: A holistic data mining approach for engineering design\". <i>Advanced Engineering Informatics<\/i> <b>28<\/b> (2): 166\u201385. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aei.2014.03.002\" target=\"_blank\">10.1016\/j.aei.2014.03.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Shape+mining%3A+A+holistic+data+mining+approach+for+engineering+design&rft.jtitle=Advanced+Engineering+Informatics&rft.aulast=Graening%2C+L.%3B+Sendhoff%2C+B.&rft.au=Graening%2C+L.%3B+Sendhoff%2C+B.&rft.date=2014&rft.volume=28&rft.issue=2&rft.pages=166%E2%80%9385&rft_id=info:doi\/10.1016%2Fj.aei.2014.03.002&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MatentzogluInfer18-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MatentzogluInfer18_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Matentzoglu, N.; Vigo, M.; Jay, C. etc. (2018). \"Inference Inspector: Improving the verification of ontology authoring actions\". <i>Journal of Web Semantics<\/i> <b>49<\/b>: 1\u201315. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.websem.2017.09.004\" target=\"_blank\">10.1016\/j.websem.2017.09.004<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Inference+Inspector%3A+Improving+the+verification+of+ontology+authoring+actions&rft.jtitle=Journal+of+Web+Semantics&rft.aulast=Matentzoglu%2C+N.%3B+Vigo%2C+M.%3B+Jay%2C+C.+etc.&rft.au=Matentzoglu%2C+N.%3B+Vigo%2C+M.%3B+Jay%2C+C.+etc.&rft.date=2018&rft.volume=49&rft.pages=1%E2%80%9315&rft_id=info:doi\/10.1016%2Fj.websem.2017.09.004&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SaltelliSens00-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SaltelliSens00_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Saltelli, A.; Chan, K.; Scott, E.M., ed. (2000). <i>Sensitivity Analysis<\/i>. Series in Probability and Statistics. Wiley. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780471998921.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Sensitivity+Analysis&rft.date=2000&rft.series=Series+in+Probability+and+Statistics&rft.pub=Wiley&rft.isbn=9780471998921&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MilesQual94-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MilesQual94_33-0\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-MilesQual94_33-1\">33.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Miles, M.B.; Huberman, A.M. (1994). <i>Qualitative Data Analysis: An Expanded Sourcebook<\/i>. SAGE Publications. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780803955400.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Qualitative+Data+Analysis%3A+An+Expanded+Sourcebook&rft.aulast=Miles%2C+M.B.%3B+Huberman%2C+A.M.&rft.au=Miles%2C+M.B.%3B+Huberman%2C+A.M.&rft.date=1994&rft.pub=SAGE+Publications&rft.isbn=9780803955400&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CorbinBasics15-34\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CorbinBasics15_34-0\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-CorbinBasics15_34-1\">34.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Corbin, J.; Strauss, A. (2015). <i>Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory<\/i> (4th ed.). SAGE Publishing. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781412997461.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Basics+of+Qualitative+Research%3A+Techniques+and+Procedures+for+Developing+Grounded+Theory&rft.aulast=Corbin%2C+J.%3B+Strauss%2C+A.&rft.au=Corbin%2C+J.%3B+Strauss%2C+A.&rft.date=2015&rft.edition=4th&rft.pub=SAGE+Publishing&rft.isbn=9781412997461&rfr_id=info:sid\/en.wikipedia.org:Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194116\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.540 seconds\nReal time usage: 0.560 seconds\nPreprocessor visited node count: 25890\/1000000\nPreprocessor generated node count: 35456\/1000000\nPost\u2010expand include size: 176066\/2097152 bytes\nTemplate argument size: 56065\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 518.409 1 - -total\n 86.69% 449.409 1 - Template:Reflist\n 73.57% 381.387 34 - Template:Citation\/core\n 63.56% 329.525 28 - Template:Cite_journal\n 13.45% 69.702 6 - Template:Cite_book\n 7.27% 37.700 1 - Template:Infobox_journal_article\n 6.96% 36.081 1 - Template:Infobox\n 6.02% 31.214 35 - Template:Citation\/identifier\n 4.76% 24.653 80 - Template:Infobox\/row\n 3.39% 17.553 35 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12291-0!*!0!!en!*!* and timestamp 20210429194115 and revision id 41260\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering\">https:\/\/www.limswiki.org\/index.php\/Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","0dd74698c3756484b9f52cadbcdc94dc_images":[],"0dd74698c3756484b9f52cadbcdc94dc_timestamp":1619725275,"2897b25895d56e2e3f8922c52ab067ea_type":"article","2897b25895d56e2e3f8922c52ab067ea_title":"Timely delivery of laboratory efficiency information, Part II: Assessing the impact of a turnaround time dashboard at a high-volume laboratory (Cassim et al. 2020)","2897b25895d56e2e3f8922c52ab067ea_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory","2897b25895d56e2e3f8922c52ab067ea_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Timely delivery of laboratory efficiency information, Part II: Assessing the impact of a turnaround time dashboard at a high-volume laboratory\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nTimely delivery of laboratory efficiency information, Part II: Assessing the impact of a turnaround time dashboard at a high-volume laboratoryJournal\n \nAfrican Journal of Laboratory MedicineAuthor(s)\n \nCassim, Naseem; Coetzee, Lindi M.; Tepper, Manfred E.; Perelson, Louella; Glencross, Deborah K.Author affiliation(s)\n \nNational Health Laboratory Service, University of the WitwatersrandPrimary contact\n \nEmail: naseem dot cassim at wits dot ac dot zaYear published\n \n2020Volume and issue\n \n9(2)Article #\n \na948DOI\n \n10.4102\/ajlm.v9i2.948ISSN\n \n2225-2010Distribution license\n \nCreative Commons Attribution 4.0 International LicenseWebsite\n \nhttps:\/\/ajlmonline.org\/index.php\/ajlm\/article\/view\/948\/1475Download\n \nhttps:\/\/ajlmonline.org\/index.php\/ajlm\/article\/download\/948\/1473 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methods \n\n3.1 Ethical considerations \n3.2 Study design and samples used \n3.3 Data extraction and turnaround time definition \n3.4 Percentage within cutoff turnaround time analysis \n3.5 75th percentile turnaround time analysis \n3.6 Root cause analysis \n\n\n4 Results \n\n4.1 Percentage of within cutoff turnaround time analysis \n4.2 75th percentile turnaround time analysis \n4.3 Root cause analysis \n\n\n5 Discussion \n\n5.1 Past approaches to improving turnaround times \n5.2 Application to African contexts \n\n\n6 Conclusion \n\n6.1 Limitations \n6.2 Lessons from the field \n\n\n7 Acknowledgements \n\n7.1 Author contributions \n7.2 Disclaimer \n7.3 Competing interests \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nBackground: In South Africa\u2019s National Health Laboratory Service, ad hoc mean turnaround time (TAT) reporting is an important indicator of performance. However, historic static TAT reporting did not assess very long or very short times. An interactive TAT dashboard was developed using the following TAT measures: (1) median, (2) 75th percentile, and (3) percentage of within cutoff TAT to allow for improved differentiation of TAT performance.\nObjectives: The objective of our study was to demonstrate increased efficiency achieved by using an interactive TAT dashboard.\nMethods: A retrospective descriptive study design was used. Creatinine TAT outcomes were reported over 122 weeks from a high-volume laboratory in Gauteng, South Africa. The percentage of within cutoff and 75th percentile TAT were analyzed and reported using Microsoft Excel. A focus group session was used to populate a cause and effect diagram.\nResults: The percentage of within cutoff TAT increased from 10% in week four to 90% and higher from week 81. The 75th percentile decreased from 10 hours in week four to under five hours from week 71. Component TAT analysis revealed that the 75th percentile testing was five hours or longer for weeks four, five and 48. The 75th percentile review TAT ranged from one hour to 15 hours. From week 41, the review TAT was under one hour.\nConclusion: Our study demonstrated that the use of an interactive TAT dashboard, coupled with good management, can dramatically improve TAT and efficiency in a high-volume laboratory.\nKeywords: turnaround time, laboratory efficiency, pathology, laboratory medicine\n\nIntroduction \nTurnaround time (TAT) is an important performance indicator of a laboratory's efficiency in delivering patient results.[1] In the South African National Health Laboratory Services, ad hoc mean TAT reports were previously produced for laboratory managers. These TAT reports assessed performance based on the National Health Laboratory Service global annual performance plan (APP) TAT cutoffs specific to individual tests.[2] Reports were provided intermittently in a static form that assessed central tendency only (i.e., the tail size was not reported) and did not allow for drilling down to access additional, more detailed information to direct meaningful corrective action (i.e., laboratory or sample-level TAT breakdown). To improve on these TAT reporting systems, Coetzee et al. used three additional measures to assess TAT efficiency: (1) median TAT, (2) 75th percentile TAT (tail size), and (3) percentage of within cutoff TAT.[3] These measures accurately assessed outliers as tail size and could be used by laboratories to address workflow issues and identify testing delays for intervention. \"Tail size\" refers to the volume of samples in a positively skewed data distribution that has a long tail to the right. These samples often have a much higher TAT value than the central tendency (median) for this data distribution. Tail size can be measured as the percentage of samples that exceed a defined TAT cutoff in hours, or as a percentile.\nInitially, the three measures described above were reported in Microsoft Excel (Redmond, Washington, United States) worksheet format from August 2016 to June 2017.[4] Thereafter, from July 2017, an interactive dashboard was developed that reported TAT data for a basket of tests using the Microstrategy Desktop (Tysons, Virginia, United States) analytics tool.[5] The previous static reports and the more recent interactive dashboard reports have seen distribution to area (province), business (district), and laboratory managers. Data can now be reviewed in the interactive dashboard reports across the provincial, district or laboratory levels through drill-down functionality, which makes it possible to slice through a data hierarchy to reveal additional details[6] contained within the aggregated data. In this way, TAT data presented can be visualized at the national, provincial, and laboratory level on the same dashboard page. The approach allows various levels of manager to drill down from a \"bird\u2019s-eye\" view of TAT performance nationally, down to the provincial or individual laboratory level.\nWithin the dashboard, TAT can be viewed for a basket of tests, including:\n\n routine hematology such as full blood count with platelet and differential testing, international normalized ratio, activated prothrombin testing, and D-dimers;\n clinical pathology testing such as urea and electrolytes, liver function testing, glucose, and cholesterol;\n microbiology testing such as HIV (HIV viral load, HIV DNA polymerase chain reaction), tuberculosis (Xpert MTB\/RIF), and syphilis (rapid plasma reagin and Treponema pallidum antibodies); and\n other disease testing methods such as cluster of differentiation 4 (CD4) testing. \nProxy marker analytes are used to assess performance of the respective matched assay panel; for example, creatinine is used as the proxy test to review urea and electrolytes performance. Each test has its own predetermined TAT determined at the local level according to the level of care, with absolute national APP cutoffs noted.\nGlobal TAT outcomes for each test are reported according to specifically stipulated, organization-determined TAT APP at the national level and are described elsewhere.[2][7] National APP cutoffs are set bearing in mind the multi-tiered service that accommodates reporting from primary health care referral to large tertiary centers that may offer emergency services, and do not necessarily reflect the respective individual, laboratory-stipulated TAT, which may be self-determined by laboratories based on their local clinical needs.\nArmed with the knowledge of TAT and which tests are identified as poor performers in the interactive dashboard, laboratory managers can identify and address areas of concern through review of the contributing causes.[8] This is achieved through root cause analysis, a method of problem solving used to identify the root causes (faults or problems) and determine the most probable underlying causes of error.[8] The ultimate aim of root cause analysis in TAT monitoring is to formulate corrective actions that either mitigate or eliminate the identified causes in order to return TAT efficiency and performance to acceptable levels.\nThe aim of this study was to report on the impact of an interactive dashboard that provides weekly information about TAT and enables laboratory and senior managers to monitor TAT, as well as identify problematic areas for corrective action. The hypothesis was that an interactive TAT dashboard delivering week-by-week information about laboratory TAT provides the impetus for continuous service review and implementation of appropriate corrective action, where required, to ensure the timeliness of laboratory reporting. Data are presented from a single busy, routinely automated clinical pathology laboratory at a large regional hospital to reveal how the described TAT dashboard served to continually highlight ongoing TAT delays for urea and electrolyte (creatinine) result reporting and, ultimately, facilitated sustained corrective action.\n\nMethods \nEthical considerations \nEthics clearance was obtained from the University of the Witwatersrand (study approval number: M1706108). No patient identifiers were extracted with data.\n\nStudy design and samples used \nA retrospective descriptive study design was used to analyze laboratory data and highlight the impact of interventions by observing trends. Qualitative focus group sessions were used to unpack the root causes of poor performance. Convenience sampling was used. For the purpose of this study, the TAT performance for creatinine testing, which had poor TAT at the start of the study, was used to demonstrate how dashboard monitoring of TAT could highlight and impact the TAT. Creatinine testing outcomes were reported with an APP cutoff of 90% within five hours.[2] Weekly TAT data\u2014from the week August 1, 2016 to August 7, 2016 (week one), through to the week November 26, 2018 to December 2, 2018 (week 122)\u2014was reviewed.\n\nData extraction and turnaround time definition \nThe data extract contained the following variables: \n\n report week ending date, e.g., October 23, 2016 (Monday to Sunday)\n laboratory name\n test method name\n TAT cutoff\n test volumes\n percentage of within cutoff TAT\n median TAT\n 75th percentile TAT\n inter-laboratory referral 75th percentile TAT\n testing 75th percentile TAT\n review 75th percentile TAT\nAll TAT 75th percentile values were reported in hours. Each week was numbered, that is, 1\u2013122. TAT data refer to total TAT (i.e., time of first registration to time of result release after review) if not otherwise specified for TAT components. All data were prepared and analyzed using Microsoft Excel (Redmond, Washington, United States).[4] The testing TAT time interval was calculated from time of registration in the testing laboratory to time of result generation on the analyzer interface. Review TAT (labeled TST-TO-RVW [test-to-review]) is the time taken by a senior technologist to review the patients\u2019 results on the laboratory information system (LIS), making sure all quality checks were adequately performed before releasing (authorizing) the patients report. The recorded time interval, that is, the review TAT, was calculated afterwards for each individual sample outcome, from the time of result generation to the time of authorization or review.\n\nPercentage within cutoff turnaround time analysis \nThe percentage of within cutoff TAT was calculated as the total number of samples meeting the organization\u2019s TAT cutoff criteria of five hours for urea and electrolytes testing divided by the total number of tests performed, expressed as a percentage, per week. The results were reported as a line chart (indicating the week number and APP cutoff of 90%). Data were segmented into three phases: (1) baseline: week 1 to 44 (week ending June 4, 2017); (2) dashboard intervention: week 45 to 63 (week ending October 15, 2017); and (3) post-intervention from week 64 to 122 (week ending December 2, 2018). The dashboard intervention period indicates the switch from using an Excel worksheet to the interactive dashboard.\n\n75th percentile turnaround time analysis \nThe 75th percentile was calculated for total TAT per week, as well as for TAT components, that is, testing and review. As tests were based out of a local hospital and not referred from surrounding laboratories, the pre-analytical TAT component was not applicable. When samples are referred, the pre-analytical TAT measures the interval (time taken to transport the sample between laboratories) from registration at the source (the laboratory where the sample was received) to the testing laboratory. Results from this analysis were plotted as 75th percentile, per testing week, for both total and component TAT.\n\nRoot cause analysis \nA root cause or Ishikawa analysis diagram was used to identify potential factors causing poor TAT performance.[9] Causes were grouped into the following headings: (1) equipment and supplies; (2) environmental; (3) rules, policies, or procedures; and (4) staff or personnel. Focus group meetings were arranged with the laboratory manager and section supervisors to identify causes and to populate the cause-and-effect diagram. A voice recorder was used to create the cause-and-effect diagram using Microsoft Visio (Redmond, Washington, United States).[4]\n\nResults \nThis laboratory performed 326,081 tests for the 2016\/2017 financial period, 341,760 tests for the 2017\/2018 financial period, and 399,538 tests for the 2018\/2019 financial period. Assuming 24\/7 operations, this equates to between 894 and 1,095 tests per day (N. Booplal, 2019, personal communication). Prior to the implementation of the interactive dashboard, weekly TAT data were extracted from the corporate data warehouse that houses LIS data within the National Health Laboratory Service. Weekly Microsoft Excel worksheets were prepared manually and distributed via email prior to the implementation of the interactive dashboard at week 45.\n\nPercentage of within cutoff turnaround time analysis \nFor the baseline phase, the percentage of TAT within the cutoff fluctuated (range: 10% to 79%) (Figure 1). During the intervention, the TAT range again fluctuated from 59% to 97%. For the post-intervention phase, the percentage of TAT with the cutoff ranged from 89% to 98%. The 90% cutoff was met for 42 consecutive weeks from week 81 to the end of the study period.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1. The percentage of within cutoff turnaround times for creatinine testing at a high-volume laboratory across 122 weeks after implementation of a weekly dashboard, Gauteng, South Africa, 2017\n\n\n\n75th percentile turnaround time analysis \nDuring the baseline phase, the total TAT 75th percentile ranged from four to 20 hours, changing to two to 10 hours for the intervention phase (Figure 2). For the post-intervention phase, the 75th percentile range was two to three hours. For testing TAT, the 75th percentile for the baseline phase ranged from two to 11 hours and changed during the intervention phase to one to six hours. In the post-intervention phase, the range was one to two hours. In the baseline phase, the 75th percentile review TAT ranged from one to 15 hours, compared to one to three hours for the intervention phase. The post-intervention phase reported a 75th percentile review TAT of one hour or less.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2. 75th percentile total turnaround time for creatinine testing at a high-volume laboratory across 122 weeks after implementation of an interactive weekly dashboard, Gauteng, South Africa, 2017\n\n\n\nRoot cause analysis \nFour major clusters of contributing causes were identified in the root cause analysis, including equipment and supplies; environmental causes; rules, policies, and procedural causes; and staff and personnel factors (Figure 3). \nWith respect to equipment and supplies, the following causes were shown to have negatively impacted TAT: (1) migration to the new platform (phased approach), (2) difficulties with procurement of laptops to facilitate after-hours off-site authorization, (3) power outages, (4) LIS bandwidth challenges, (5) LIS upgrades, and (6) reagent or stock procurement. \nInsufficient air conditioning and water leaks from the ceiling were highlighted as causative environmental factors. \nProblems related to rules, policies, and procedures included (1) middleware having to be configured, tested, and amended due to the changes brought about by a phased approach, and (2) a substantial workload being transferred from a nearby laboratory without provision of additional testing or staffing capacity. \nFor staff and personnel considerations, the following were identified: (1) after-hours authorization delay (by pathologists); (2) industrial action leading to delays; (3) paediatric and low-volume samples requiring manual processing, causing bottlenecks in the workflow; (4) staff constraints (in terms of insufficient staff to manage the benches); (5) additional training of staff being required for new procedures and processes for the platform testing changes implemented; (6) first-line manual sample preparation being needed to enable sample testing prior to full automation; and (7) training for the new platforms provided occurring on-site, while at the same time staff were also required to attend training off-site, leaving benches poorly staffed during training periods.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 3. Root cause analysis diagram developed in conjunction with the laboratory manager at a high-volume laboratory, Gauteng, South Africa, 2017\n\n\n\nDiscussion \nIn this study, it was hypothesized that the application of appropriate corrective action, guided by an interactive TAT dashboard indicating the proportion of samples within stipulated TAT cutoffs and tail size (outliers), would result in improved performance.[1][10] This was based on the assumption that delivering an interactive TAT dashboard indicating outlier performance on its own would not result in improvement. Good laboratory management and response with appropriate corrective action is the key catalyst to deliver a sustained quality of reporting and ensure the continual TAT improvement of performance over time.[11]\nAll laboratories typically adhere to a quality management system (QMS) that is used to assess laboratory quality from the pre-analytic phase through the testing and reviewing processes. A QMS is defined as a set of coordinated activities that direct and control a laboratory with regard to quality.[12] All aspects of the laboratory operation, including the organizational structure, processes, and procedures need to be addressed by the QMS to assure quality.[12] Laboratory quality can be defined in terms of accuracy, reliability, and timeliness (i.e., TAT).[12] One of the key practices for continual improvement is the management review meeting, allowing the laboratory an opportunity to review annual performance as set out in the QMS. The management review cycle involves planning, implementing, checking, and acting on a quarterly basis to address shortfalls identified to affect continual improvement.[12]\nIn this study, we reveal how the introduction of a TAT dashboard enabled senior management of the laboratory in question to assess TAT performance for a particular battery of tests that had not met the stipulated TAT cutoff (greater than 65% of results were outside of the stipulated TAT cutoff). Upon introduction of the dashboard, several areas of concern were immediately identified, including pre-analytical, analytical, and post-analytical factors. With respect to component TAT, assessing specifically the timeframes of \"registration to testing\" and \"testing to release of the report,\" TAT delays were attributable to delays of review during weeks one to 41, with further delays caused by testing (instrument) interruptions during weeks four and 48. The root cause analysis revealed several contributing factors, categorized as equipment and supplies; environmental; rules, policies, and procedures; and personnel or staffing issues. Specifically, the introduction of an auto-review rule process played an important role in improving TAT cutoff. A similarly placed high-volume core laboratory in Canada also reported that the implementation of a series of lean approaches in their busy laboratory, including automation and auto-review rules, proved effective for more efficiently managing substantial volumes of samples while meeting TAT cutoffs.[13]\nSeveral important lessons learned and documented by the study laboratory could serve as a template for outreach training to help other public sector laboratories achieve similar TAT performance improvements and establish the practices adopted at this site. Firstly, the importance of the need to collate and actively review real-time information about TAT\u2014including components of TAT, in ensuring overall timely reporting in laboratories\u2014was understood and confirmed. Secondly, the value of vertical audits was demonstrated. Vertical audits assisted in understanding what contributed to delayed TAT, and specific focus on outlier samples and vertical audits directed subsequent meaningful corrective action. In line with the requirement of ongoing improvement of service delivery, a weekly \"results for action\" statement was developed and found to be useful to deliver specific information at the sample or episode level. Such reports, while getting the attention of senior management, could be directed to relevant managers to highlight specific problematic areas and guide the focus of managers\u2019 attention to the investigation of true TAT outliers or exceptions. Such investigation (with solutions) of specifically identified problem areas could yield practical and advantageous outcomes, not only solving the issues at hand, but more widely having a positive impact on overall service delivery improvement. \nThe final lesson learned revolved around the importance of documenting and following through on corrective actions as part of the QMS. This ensures that corrective actions taken have consequences and are sustained. In the services review presented here, the week-by-week reminders of outlying TAT were a constant cue that solutions implemented had not been effective. Re-evaluation and re-assessment allowed for streamlined processes to be considered when initial corrective actions had failed. Also highlighted was the importance of conducting a root cause analysis using cause-and-effect diagrams, to tease out and understand all aspects of errors and any contributing factors that may lead to delays in TAT. It is also important to point out that although a corrective action may be resolved with a single intervention, more frequently, corrective action proves to be a multi-step process to identify possible solutions and alternatives. Once implemented, these corrective steps require consistent monitoring and evaluation for sustained impact. Here, the information provided by the dashboard offered objective evidence of identified issues that could be documented and presented to senior managers from month to month and at the annual management meeting; that, in due course, enabled corrective action planning and the facilitated, necessary mandates to affect better service.\n\nPast approaches to improving turnaround times \nOver the years, there have been multiple approaches to monitoring TAT reporting with the aim of improving TAT and, in turn, patient care. Approaches include identifying outliers, implementing Lean Sigma Six, and performing process mapping.[1][14][15][16][17] One of the earliest approaches described was the identification of TAT outliers.[1] In 2005, Holland et al. reported that the average length of stay in the emergency department across 11 hospitals correlated significantly with the percentage of outliers.[15] In 2007, Hawkins defined outliers as the TAT of samples that exceeded the institution\u2019s agreed TAT cutoffs.[1] Outliers could also be defined as the tail size given the skewed TAT distribution. Therefore, the mean and confidence interval are not appropriate measures for assessing TAT performance. In 2018, Coetzee et al. used the 75th percentile and the percentage of samples within the cutoff TAT to identify outliers at the laboratory and test levels.[10] The development of routine monitoring systems to identify laboratories with a long tail size enabled focused interventions to proactively resolve poor service delivery.[10]\nMany laboratories have implemented Lean Six Sigma approaches to improve TAT performance.[13] \"Lean\" is defined as a continual improvement process consisting of technical tools and management methods.[16] One of the aims of a lean approach is to find and eliminate waste.[16] For example, waste may be introduced by the layout of a laboratory or by poor process design.[16] Long distances between the receiving office and testing laboratory could also, for example, encourage staff to move samples in batches resulting in TAT delays.[16] In 2008, Graban and Padgett reported that the introduction of lean approaches for troponin testing resulted in substantial TAT improvements and averted a proposed point-of-care testing implementation.[16] A 2012 report by Stapleton and Sammond showed how a lean approach implemented over a three-year period in the laboratory resulted in TAT reductions for all emergency tests by implementing both workflow improvements and a dedicated emergency bench.[17]\nAnother approach to improve TAT was reported by Barakauskas et al. in 2016, involving the use of LIS time stamps, direct observations, and discussions with staff to construct various value stream and process maps for immunosuppressant drug level testing.[14] The value stream map identified process bottlenecks that were addressed.[14] The process map was reported in columns to represent the major groups of personnel and locations from the health care worker to the reference testing laboratory, with the sequence of events and steps involved illustrated in a vertical direction.[14] Bottlenecks were also identified in the process map to plan improvement initiatives, e.g., the use of emergency bags.[14]\nUltimately, the aim of any of the approaches described above is to improve TAT performance and, thus, patient care. Although TAT is especially important for emergency tests that have very short TAT cutoffs, it is equally important to set cutoffs for other, less urgent tests to ensure that respective test results are received by attending physicians in a timely fashion to affect appropriate patient care, an important factor in assuring both the quality of care and the cost-effective use of hospital services.[18] Aside from patient care, TAT delays also have the potential to waste valuable health resources caused by duplicate test requests, thereby increasing public health expenditure.[18] In summary, the multiple approaches to improving TAT performance across all laboratory tests play an important role in improving the quality of patient care.\n\nApplication to African contexts \nFigure 4 describes an approach that would make it possible for laboratories in low- and middle-income countries to collate the data required to develop the TAT reporting described in our study. In the first instance, laboratories would require a basic LIS that could generate weekly data extracts as described in Figure 4, using open-source database software such as Microsoft SQL Server (Express, Redmond, California, United States).[19] Equally, MySQL[20], Firebird[21], or CUBRID[22] could be deployed to generate the aggregate data described. Training is freely accessible via the internet for these software packages. There are multiple free online courses by providers via platforms such as edX, where one can learn how to both develop and query a structured query language (SQL) database.[23][24] Any of these software packages could be implemented on a local desktop or in a server environment, depending on the data volume, with very basic query tools using SQL commands, making it possible to develop dashboard tools using the step-by-step building blocks approach described in Figure 4.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 4. Integration of laboratory information system specimen-level turnaround time data from multiple laboratories into a single, aggregated structured query language database for development of an interactive dashboard, Gauteng, South Africa, 2017\n\n\n\nCross-country collaboration and sharing of resources could play an important role in securing already developed dashboard tools for other African countries. A multi-country approach could reduce overall costs and effort. For example, a single TAT dashboard could be developed for the Southern African Development Community to ensure accessibility and provide scalability. To secure the system and provide confidentiality, each country could have access to their own data using data access privileges. The benefit of this approach is that after the methods, systems, and dashboards are developed, it is easy to extend these developments to other countries with minimal additional cost. The only additional effort required at the country level would be to collate and share the data extracts with the umbrella organizers.\n\nConclusion \nIn summary, this study demonstrated that an interactive TAT dashboard that reports appropriate TAT parameters, applied in the context of a QMS and coupled with proactive and diligent management, can accurately identify outliers and lead to appropriate corrective action and sustained timely laboratory reporting.\n\nLimitations \nOnly LIS data were used for our study. Without a laboratory specimen tracking system, it is not possible to report end-to-end TAT. The implementation of an end-to-end tracking system from the time of venesection to delivery into a laboratory, additionally integrated into a TAT dashboard, could provide valuable supplementary date and time values to allow for an extended TAT efficiency review.\n\nLessons from the field \nOur experiences in the field allow us provide a few lessons learned:\n\n A weekly interactive TAT dashboard enables reporting of appropriate TAT paramaters and respective outcomes by confirming ongoing quality and timely reporting, as well as identifying outlying TAT that may require appropriate corrective action.\n TAT data can be collected at the laboratory, local network, or national level. A dashboard that includes aggregated and local level data, with a drill-down function, allows hierarchical review of the data, such that both higher-level managers and laboratory managers are able to view the same data, but at different levels appropriate for their respective level of responsibility.\n Continuously collating and analyzing the data and presenting TAT information in a user-friendly, visual dashboard format allows for immediate attention to be focused on outlying sites and areas.\n Visibility and transparency of TAT data and outcomes to all levels of management provide an incentive (with repeated peer or organization pressure, if consistently outside of TAT) to act on poor performance.\n A quality management system requires active input, monitoring, and appropriate action where needed. The presentation of information does not necessarily confer good performance or the meeting of TAT cutoffs. A dashboard, such as that presented here, is merely a tool. Proactive, consistent, and diligent review of TAT data presented in a dashboard is required to facilitate meaningful improvement and corrective action. An auto-review rule implemented for a specific test or battery of tests on the LIS has the potential to reduce TAT by acting to reduce the workload for senior staff through automatic review of predominantly normal results. With the auto-review implemented, senior staff effectively use their time and reserve the resultant review only for samples that fail to meet the auto-review rule, e.g., as with delta check failures.\nAcknowledgements \nThe authors thank area, business, and laboratory managers in Gauteng for their participation and input.\n\nAuthor contributions \nD.K.G. supervised the study by providing leadership and oversight as the project leader. N.C., M.E.E.T. and L.M.C. designed the study, developed the methodology and conducted the research. M.E.E.T. extracted the data. N.C. conducted the data analysis. L.P. provided content for the cause and effect diagram and provided details on the interventions implemented. D.K.G. provided editorial comments and technical input. All authors reviewed the results and contributed to the manuscript development.\n\nDisclaimer \nThe authors declare that the views expressed in the submitted article are their own and not the official position of any institution or funder.\n\nCompeting interests \nThe authors declare no conflict of interest. The authors further declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.\n\nReferences \n\n\n\u2191 1.0 1.1 1.2 1.3 1.4 Hawkins, R.C. (2007). \"Laboratory turnaround time\". The Clinical Biochemist 28 (4): 179\u201394. PMC PMC2282400. PMID 18392122. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2282400 .   \n\n\u2191 2.0 2.1 2.2 National Health Laboratory Service (2018) (PDF). National Health Laboratory Service Annual Report 2017\/18. National Health Laboratory Service. ISBN 9780621457018. https:\/\/nationalgovernment.co.za\/entity_annual\/1714\/2018-national-health-laboratory-service-(nhls)-annual-report.pdf .   \n\n\u2191 Coetzee, L.M.; Cassim, N.; Tepper, M.E.E. et al. (2018). \"The importance of reporting individual weekly laboratory turn-around-time (TAT) to identify outliers and underperformance masked during global annual TAT review\". Proceedings of the African Society for Laboratory Medicine Conference 2018. https:\/\/www.researchgate.net\/publication\/329610644 . \"Poster ID: PS-2.3b-070\"   \n\n\u2191 4.0 4.1 4.2 \"Apps and services\". Microsoft. https:\/\/www.microsoft.com\/en-za\/microsoft-365\/products-apps-services . Retrieved 12 March 2019 .   \n\n\u2191 \"Download MicroStrategy Desktop\". MicroStrategy. https:\/\/www.microstrategy.com\/en\/get-started\/desktop . Retrieved 03 December 2018 .   \n\n\u2191 Microsoft. \"Drill mode in a visual in Power BI\". https:\/\/docs.microsoft.com\/en-us\/power-bi\/consumer\/end-user-drill . Retrieved 12 December 2018 .   \n\n\u2191 Cassim, N.; Tepper, M.E.; Coetzee, L.M.; Glencross, D.K. (2020). \"Timely delivery of laboratory efficiency information, Part I: Developing an interactive turnaround time dashboard at a high-volume laboratory\". African Journal of Laboratory Medicine 9 (2): a947. doi:10.4102\/ajlm.v9i2.947. PMC PMC7203318. PMID 32391244. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7203318 .   \n\n\u2191 8.0 8.1 Khan, K.. \"Root Cause Analysis (RCA) of Prolonged Laboratory Turnaround Time in a Tertiary Care Set Up\". Journal of Clinical and Diagnostic Research 8 (4): FC05\u2013FC08. doi:10.7860\/JCDR\/2014\/7269.4255. PMC PMC4064846. PMID 24959450. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4064846 .   \n\n\u2191 Ishikawa, K. (1976). Guide to Quality Control (2nd Revised ed.). Asian Productivity Organization. ISBN 9283310365. https:\/\/archive.org\/details\/guidetoqualityco00ishi\/page\/n7\/mode\/2up .   \n\n\u2191 10.0 10.1 10.2 Coetzee, L.-M.; Cassim, N.; Glencross, D.K. (2018). \"Using laboratory data to categorise CD4 laboratory turn-around-time performance across a national programme\". African Journal of Laboratory Medicine 7 (1): 665. doi:10.4102\/ajlm.v7i1.665. PMC PMC6111574. PMID 30167387. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6111574 .   \n\n\u2191 Ezzelle, J.; Rodriguez-Chavez, I.R.; Darden, J.M. et al. (2008). \"Guidelines on good clinical laboratory practice: bridging operations between research and clinical research laboratories\". Journal of Pharmaceutical and Biomedical Analysis 46 (1): 18\u201329. doi:10.1016\/j.jpba.2007.10.010. PMC PMC2213906. PMID 18037599. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2213906 .   \n\n\u2191 12.0 12.1 12.2 12.3 World Health Organization (2011) (PDF). Laboratory Quality Management System Handbook. World Health Organization. ISBN 9789241548274. https:\/\/www.who.int\/ihr\/publications\/lqms_en.pdf .   \n\n\u2191 13.0 13.1 Lou, A.H.; Elnenaei, M.O.; Sadek, I. et al. (2017). \"Multiple pre- and post-analytical lean approaches to the improvement of the laboratory turnaround time in a large core laboratory\". Clinical Biochemistry 50 (15): 864\u201369. doi:10.1016\/j.clinbiochem.2017.04.019. PMID 28457964.   \n\n\u2191 14.0 14.1 14.2 14.3 14.4 Barakauskas, V.E.; Bradshaw, T.A.; Smith, L.D. et al. (2016). \"Process Optimization to Improve Immunosuppressant Drug Testing Turnaround Time\". American Journal of Clinical Pathology 146 (2): 182\u201390. doi:10.1093\/ajcp\/aqw087. PMID 27453440.   \n\n\u2191 15.0 15.1 Holland, L.L.; Smith, L.L.; Blick, K.E. (2005). \"Reducing laboratory turnaround time outliers can reduce emergency department patient length of stay: An 11-hospital study\". American Journal of Clinical Pathology 124 (5): 672\u20134. doi:10.1309\/E9QP-VQ6G-2FBV-MJ3B. PMID 16203280.   \n\n\u2191 16.0 16.1 16.2 16.3 16.4 16.5 Graban, M.; Padgett, S. (2008). \"Lean Laboratories: Competing with Methods From Toyota\". Laboratory Medicine 39 (11): 645\u201348. doi:10.1309\/LMX0LEMR7R0USKUM.   \n\n\u2191 17.0 17.1 Stapleton, K.; Sammond, D. (2012). \"Improved Laboratory Stat Test Turnaround Time Using Lean Six Sigma\". American Journal of Clinical Pathology 138 (Suppl. 2): A184. doi:10.1093\/ajcp\/138.suppl2.36.   \n\n\u2191 18.0 18.1 Winkelman, J.W.; Tanasijevic, M.J.; Wybenga, D.R. et al. (1997). \"How fast is fast enough for clinical laboratory turnaround time? Measurement of the interval between result entry and inquiries for reports\". American Journal of Clinical Pathology 108 (4): 400\u20135. doi:10.1093\/ajcp\/108.4.400. PMID 9322592.   \n\n\u2191 Microsoft. \"SQL Server 2019\". https:\/\/www.microsoft.com\/en-us\/sql-server\/sql-server-2019 . Retrieved 10 April 2019 .   \n\n\u2191 Oracle Corporation. \"MySQL Community Edition\". https:\/\/www.mysql.com\/products\/community\/ . Retrieved 03 February 2019 .   \n\n\u2191 Firebird Project. \"Firebird RDBMS\". https:\/\/firebirdsql.org\/en\/firebird-rdbms\/ . Retrieved 10 April 2019 .   \n\n\u2191 CUBRID Foundation. \"CUBRID\". https:\/\/www.cubrid.org\/ . Retrieved 10 April 2019 .   \n\n\u2191 Microsoft. \"Querying Data with Transact-SQL\". edX. https:\/\/www.edx.org\/course\/querying-data-with-transact-sql-2 . Retrieved 03 April 2019 .   \n\n\u2191 Microsoft. \"Developing SQL Databases\". edX. Archived from the original on 20 May 2020. https:\/\/web.archive.org\/web\/20200502074237\/http:\/\/www.edx.org\/course\/developing-sql-databases . Retrieved 03 April 2019 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The original citation number two (2017-2018 NHLS Annual Report) was dead; an alternately hosted version was found and used for this version. The original cites Wikipedia concerning the Ishikawa diagram; as using Wikipedia as a citation is generally frowned upon, a substitute citation for Ishikawa's original Guide to Quality Control was used for this version. The original article accidentally mixed up citations 19 and 20; this was corrected for this version. The citation URL for the SQL Databases edX course is dead; an archived version of the URL is used for this version. The original article used an image for lessons learned in the field; this version skips the image and replaces it with the image text.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\">https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on laboratory informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 24 January 2021, at 20:21.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 230 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","2897b25895d56e2e3f8922c52ab067ea_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Timely_delivery_of_laboratory_efficiency_information_Part_II_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Timely delivery of laboratory efficiency information, Part II: Assessing the impact of a turnaround time dashboard at a high-volume laboratory<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: In South Africa\u2019s National Health Laboratory Service, <i>ad hoc<\/i> mean <a href=\"https:\/\/en.wikipedia.org\/wiki\/Turnaround_time\" class=\"extiw wiki-link\" title=\"wikipedia:Turnaround time\" data-key=\"ddf25c1267aa8e0b4ebe6301570d3bd4\">turnaround time<\/a> (TAT) <a href=\"https:\/\/www.limswiki.org\/index.php\/Reporting\" title=\"Reporting\" class=\"mw-disambig wiki-link\" data-key=\"c83685a5b9154b511ee113bbffedb2e5\">reporting<\/a> is an important indicator of performance. However, historic static TAT reporting did not assess very long or very short times. An interactive TAT dashboard was developed using the following TAT measures: (1) median, (2) 75th percentile, and (3) percentage of within cutoff TAT to allow for improved differentiation of TAT performance.\n<\/p><p><b>Objectives<\/b>: The objective of our study was to demonstrate increased efficiency achieved by using an interactive TAT dashboard.\n<\/p><p><b>Methods<\/b>: A retrospective descriptive study design was used. Creatinine TAT outcomes were reported over 122 weeks from a high-volume <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> in Gauteng, South Africa. The percentage of within cutoff and 75th percentile TAT were analyzed and reported using Microsoft Excel. A focus group session was used to populate a cause and effect diagram.\n<\/p><p><b>Results<\/b>: The percentage of within cutoff TAT increased from 10% in week four to 90% and higher from week 81. The 75th percentile decreased from 10 hours in week four to under five hours from week 71. Component TAT analysis revealed that the 75th percentile testing was five hours or longer for weeks four, five and 48. The 75th percentile review TAT ranged from one hour to 15 hours. From week 41, the review TAT was under one hour.\n<\/p><p><b>Conclusion<\/b>: Our study demonstrated that the use of an interactive TAT dashboard, coupled with good management, can dramatically improve TAT and efficiency in a high-volume laboratory.\n<\/p><p><b>Keywords<\/b>: turnaround time, laboratory efficiency, pathology, laboratory medicine\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Turnaround_time\" class=\"extiw wiki-link\" title=\"wikipedia:Turnaround time\" data-key=\"ddf25c1267aa8e0b4ebe6301570d3bd4\">Turnaround time<\/a> (TAT) is an important performance indicator of a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a>'s efficiency in delivering patient results.<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-0\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup> In the South African National Health Laboratory Services, <i>ad hoc<\/i> mean TAT reports were previously produced for laboratory managers. These TAT reports assessed performance based on the National Health Laboratory Service global annual performance plan (APP) TAT cutoffs specific to individual tests.<sup id=\"rdp-ebb-cite_ref-NHLSAnn1819_2-0\" class=\"reference\"><a href=\"#cite_note-NHLSAnn1819-2\">[2]<\/a><\/sup> Reports were provided intermittently in a static form that assessed central tendency only (i.e., the tail size was not reported) and did not allow for drilling down to access additional, more detailed <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> to direct meaningful corrective action (i.e., laboratory or sample-level TAT breakdown). To improve on these TAT reporting systems, Coetzee <i>et al.<\/i> used three additional measures to assess TAT efficiency: (1) median TAT, (2) 75th percentile TAT (tail size), and (3) percentage of within cutoff TAT.<sup id=\"rdp-ebb-cite_ref-CoetzeeTheImport18_3-0\" class=\"reference\"><a href=\"#cite_note-CoetzeeTheImport18-3\">[3]<\/a><\/sup> These measures accurately assessed outliers as tail size and could be used by laboratories to address workflow issues and identify testing delays for intervention. \"Tail size\" refers to the volume of <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">samples<\/a> in a positively skewed data distribution that has a long tail to the right. These samples often have a much higher TAT value than the central tendency (median) for this data distribution. Tail size can be measured as the percentage of samples that exceed a defined TAT cutoff in hours, or as a percentile.\n<\/p><p>Initially, the three measures described above were reported in Microsoft Excel (Redmond, Washington, United States) worksheet format from August 2016 to June 2017.<sup id=\"rdp-ebb-cite_ref-Microsoft_4-0\" class=\"reference\"><a href=\"#cite_note-Microsoft-4\">[4]<\/a><\/sup> Thereafter, from July 2017, an interactive dashboard was developed that reported TAT data for a basket of tests using the Microstrategy Desktop (Tysons, Virginia, United States) analytics tool.<sup id=\"rdp-ebb-cite_ref-MicroStratDesk_5-0\" class=\"reference\"><a href=\"#cite_note-MicroStratDesk-5\">[5]<\/a><\/sup> The previous static reports and the more recent interactive dashboard reports have seen distribution to area (province), business (district), and laboratory managers. Data can now be reviewed in the interactive dashboard reports across the provincial, district or laboratory levels through drill-down functionality, which makes it possible to slice through a data hierarchy to reveal additional details<sup id=\"rdp-ebb-cite_ref-MicrosoftDrill_6-0\" class=\"reference\"><a href=\"#cite_note-MicrosoftDrill-6\">[6]<\/a><\/sup> contained within the aggregated data. In this way, TAT data presented can be visualized at the national, provincial, and laboratory level on the same dashboard page. The approach allows various levels of manager to drill down from a \"bird\u2019s-eye\" view of TAT performance nationally, down to the provincial or individual laboratory level.\n<\/p><p>Within the dashboard, TAT can be viewed for a basket of tests, including:\n<\/p>\n<ul><li> routine <a href=\"https:\/\/www.limswiki.org\/index.php\/Hematology\" title=\"Hematology\" class=\"wiki-link\" data-key=\"de8b49c7b0be3cec33af362e763b9b0c\">hematology<\/a> such as full blood count with platelet and differential testing, international normalized ratio, activated prothrombin testing, and D-dimers;<\/li>\n<li> <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_pathology\" title=\"Clinical pathology\" class=\"wiki-link\" data-key=\"9b0b91aa13ae0209e073db6f30493daf\">clinical pathology<\/a> testing such as urea and electrolytes, liver function testing, glucose, and cholesterol;<\/li>\n<li> microbiology testing such as HIV (HIV viral load, HIV DNA polymerase chain reaction), tuberculosis (Xpert MTB\/RIF), and syphilis (rapid plasma reagin and <i>Treponema pallidum<\/i> antibodies); and<\/li>\n<li> other disease testing methods such as cluster of differentiation 4 (CD4) testing. <\/li><\/ul>\n<p>Proxy marker analytes are used to assess performance of the respective matched assay panel; for example, creatinine is used as the proxy test to review urea and electrolytes performance. Each test has its own predetermined TAT determined at the local level according to the level of care, with absolute national APP cutoffs noted.\n<\/p><p>Global TAT outcomes for each test are reported according to specifically stipulated, organization-determined TAT APP at the national level and are described elsewhere.<sup id=\"rdp-ebb-cite_ref-NHLSAnn1819_2-1\" class=\"reference\"><a href=\"#cite_note-NHLSAnn1819-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CassimFac1_20_7-0\" class=\"reference\"><a href=\"#cite_note-CassimFac1_20-7\">[7]<\/a><\/sup> National APP cutoffs are set bearing in mind the multi-tiered service that accommodates reporting from primary health care referral to large tertiary centers that may offer emergency services, and do not necessarily reflect the respective individual, laboratory-stipulated TAT, which may be self-determined by laboratories based on their local clinical needs.\n<\/p><p>Armed with the knowledge of TAT and which tests are identified as poor performers in the interactive dashboard, laboratory managers can identify and address areas of concern through review of the contributing causes.<sup id=\"rdp-ebb-cite_ref-KhanRoot14_8-0\" class=\"reference\"><a href=\"#cite_note-KhanRoot14-8\">[8]<\/a><\/sup> This is achieved through root cause analysis, a method of problem solving used to identify the root causes (faults or problems) and determine the most probable underlying causes of error.<sup id=\"rdp-ebb-cite_ref-KhanRoot14_8-1\" class=\"reference\"><a href=\"#cite_note-KhanRoot14-8\">[8]<\/a><\/sup> The ultimate aim of root cause analysis in TAT monitoring is to formulate corrective actions that either mitigate or eliminate the identified causes in order to return TAT efficiency and performance to acceptable levels.\n<\/p><p>The aim of this study was to report on the impact of an interactive dashboard that provides weekly information about TAT and enables laboratory and senior managers to monitor TAT, as well as identify problematic areas for corrective action. The hypothesis was that an interactive TAT dashboard delivering week-by-week information about laboratory TAT provides the impetus for continuous service review and implementation of appropriate corrective action, where required, to ensure the timeliness of laboratory reporting. Data are presented from a single busy, routinely automated clinical pathology laboratory at a large regional <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> to reveal how the described TAT dashboard served to continually highlight ongoing TAT delays for urea and electrolyte (creatinine) result reporting and, ultimately, facilitated sustained corrective action.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Ethical_considerations\">Ethical considerations<\/span><\/h3>\n<p>Ethics clearance was obtained from the University of the Witwatersrand (study approval number: M1706108). No patient identifiers were extracted with data.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Study_design_and_samples_used\">Study design and samples used<\/span><\/h3>\n<p>A retrospective descriptive study design was used to analyze laboratory data and highlight the impact of interventions by observing trends. Qualitative focus group sessions were used to unpack the root causes of poor performance. Convenience sampling was used. For the purpose of this study, the TAT performance for creatinine testing, which had poor TAT at the start of the study, was used to demonstrate how dashboard monitoring of TAT could highlight and impact the TAT. Creatinine testing outcomes were reported with an APP cutoff of 90% within five hours.<sup id=\"rdp-ebb-cite_ref-NHLSAnn1819_2-2\" class=\"reference\"><a href=\"#cite_note-NHLSAnn1819-2\">[2]<\/a><\/sup> Weekly TAT data\u2014from the week August 1, 2016 to August 7, 2016 (week one), through to the week November 26, 2018 to December 2, 2018 (week 122)\u2014was reviewed.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_extraction_and_turnaround_time_definition\">Data extraction and turnaround time definition<\/span><\/h3>\n<p>The data extract contained the following variables: \n<\/p>\n<ul><li> report week ending date, e.g., October 23, 2016 (Monday to Sunday)<\/li>\n<li> laboratory name<\/li>\n<li> test method name<\/li>\n<li> TAT cutoff<\/li>\n<li> test volumes<\/li>\n<li> percentage of within cutoff TAT<\/li>\n<li> median TAT<\/li>\n<li> 75th percentile TAT<\/li>\n<li> inter-laboratory referral 75th percentile TAT<\/li>\n<li> testing 75th percentile TAT<\/li>\n<li> review 75th percentile TAT<\/li><\/ul>\n<p>All TAT 75th percentile values were reported in hours. Each week was numbered, that is, 1\u2013122. TAT data refer to total TAT (i.e., time of first registration to time of result release after review) if not otherwise specified for TAT components. All data were prepared and analyzed using Microsoft Excel (Redmond, Washington, United States).<sup id=\"rdp-ebb-cite_ref-Microsoft_4-1\" class=\"reference\"><a href=\"#cite_note-Microsoft-4\">[4]<\/a><\/sup> The testing TAT time interval was calculated from time of registration in the testing laboratory to time of result generation on the analyzer interface. Review TAT (labeled <code>TST-TO-RVW<\/code> [test-to-review]) is the time taken by a senior technologist to review the patients\u2019 results on the <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS), making sure all quality checks were adequately performed before releasing (authorizing) the patients report. The recorded time interval, that is, the review TAT, was calculated afterwards for each individual sample outcome, from the time of result generation to the time of authorization or review.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Percentage_within_cutoff_turnaround_time_analysis\">Percentage within cutoff turnaround time analysis<\/span><\/h3>\n<p>The percentage of within cutoff TAT was calculated as the total number of samples meeting the organization\u2019s TAT cutoff criteria of five hours for urea and electrolytes testing divided by the total number of tests performed, expressed as a percentage, per week. The results were reported as a line chart (indicating the week number and APP cutoff of 90%). Data were segmented into three phases: (1) baseline: week 1 to 44 (week ending June 4, 2017); (2) dashboard intervention: week 45 to 63 (week ending October 15, 2017); and (3) post-intervention from week 64 to 122 (week ending December 2, 2018). The dashboard intervention period indicates the switch from using an Excel worksheet to the interactive dashboard.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"75th_percentile_turnaround_time_analysis\">75th percentile turnaround time analysis<\/span><\/h3>\n<p>The 75th percentile was calculated for total TAT per week, as well as for TAT components, that is, testing and review. As tests were based out of a local hospital and not referred from surrounding laboratories, the pre-analytical TAT component was not applicable. When samples are referred, the pre-analytical TAT measures the interval (time taken to transport the sample between laboratories) from registration at the source (the laboratory where the sample was received) to the testing laboratory. Results from this analysis were plotted as 75th percentile, per testing week, for both total and component TAT.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Root_cause_analysis\">Root cause analysis<\/span><\/h3>\n<p>A root cause or Ishikawa analysis diagram was used to identify potential factors causing poor TAT performance.<sup id=\"rdp-ebb-cite_ref-IshikawaGuide76_9-0\" class=\"reference\"><a href=\"#cite_note-IshikawaGuide76-9\">[9]<\/a><\/sup> Causes were grouped into the following headings: (1) equipment and supplies; (2) environmental; (3) rules, policies, or procedures; and (4) staff or personnel. Focus group meetings were arranged with the laboratory manager and section supervisors to identify causes and to populate the cause-and-effect diagram. A voice recorder was used to create the cause-and-effect diagram using Microsoft Visio (Redmond, Washington, United States).<sup id=\"rdp-ebb-cite_ref-Microsoft_4-2\" class=\"reference\"><a href=\"#cite_note-Microsoft-4\">[4]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>This laboratory performed 326,081 tests for the 2016\/2017 financial period, 341,760 tests for the 2017\/2018 financial period, and 399,538 tests for the 2018\/2019 financial period. Assuming 24\/7 operations, this equates to between 894 and 1,095 tests per day (N. Booplal, 2019, personal communication). Prior to the implementation of the interactive dashboard, weekly TAT data were extracted from the corporate <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_warehouse\" title=\"Data warehouse\" class=\"wiki-link\" data-key=\"ca506499cdf544371c0a0d549ff0e9ee\">data warehouse<\/a> that houses LIS data within the National Health Laboratory Service. Weekly Microsoft Excel worksheets were prepared manually and distributed via email prior to the implementation of the interactive dashboard at week 45.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Percentage_of_within_cutoff_turnaround_time_analysis\">Percentage of within cutoff turnaround time analysis<\/span><\/h3>\n<p>For the baseline phase, the percentage of TAT within the cutoff fluctuated (range: 10% to 79%) (Figure 1). During the intervention, the TAT range again fluctuated from 59% to 97%. For the post-intervention phase, the percentage of TAT with the cutoff ranged from 89% to 98%. The 90% cutoff was met for 42 consecutive weeks from week 81 to the end of the study period.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Cassim_AfricanJLabMed2020_9-2-b.jpg\" class=\"image wiki-link\" data-key=\"99b5dbd5d46e489a1e587fe76253d61d\"><img alt=\"Fig1 Cassim AfricanJLabMed2020 9-2-b.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ee\/Fig1_Cassim_AfricanJLabMed2020_9-2-b.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1.<\/b> The percentage of within cutoff turnaround times for creatinine testing at a high-volume laboratory across 122 weeks after implementation of a weekly dashboard, Gauteng, South Africa, 2017<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"75th_percentile_turnaround_time_analysis_2\">75th percentile turnaround time analysis<\/span><\/h3>\n<p>During the baseline phase, the total TAT 75th percentile ranged from four to 20 hours, changing to two to 10 hours for the intervention phase (Figure 2). For the post-intervention phase, the 75th percentile range was two to three hours. For testing TAT, the 75th percentile for the baseline phase ranged from two to 11 hours and changed during the intervention phase to one to six hours. In the post-intervention phase, the range was one to two hours. In the baseline phase, the 75th percentile review TAT ranged from one to 15 hours, compared to one to three hours for the intervention phase. The post-intervention phase reported a 75th percentile review TAT of one hour or less.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Cassim_AfricanJLabMed2020_9-2-b.jpg\" class=\"image wiki-link\" data-key=\"24f5e3fc4f0c74bdcbb3775e533bbeb7\"><img alt=\"Fig2 Cassim AfricanJLabMed2020 9-2-b.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/fb\/Fig2_Cassim_AfricanJLabMed2020_9-2-b.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2.<\/b> 75th percentile total turnaround time for creatinine testing at a high-volume laboratory across 122 weeks after implementation of an interactive weekly dashboard, Gauteng, South Africa, 2017<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Root_cause_analysis_2\">Root cause analysis<\/span><\/h3>\n<p>Four major clusters of contributing causes were identified in the root cause analysis, including equipment and supplies; environmental causes; rules, policies, and procedural causes; and staff and personnel factors (Figure 3). \n<\/p><p>With respect to equipment and supplies, the following causes were shown to have negatively impacted TAT: (1) migration to the new platform (phased approach), (2) difficulties with procurement of laptops to facilitate after-hours off-site authorization, (3) power outages, (4) LIS bandwidth challenges, (5) LIS upgrades, and (6) reagent or stock procurement. \n<\/p><p>Insufficient air conditioning and water leaks from the ceiling were highlighted as causative environmental factors. \n<\/p><p>Problems related to rules, policies, and procedures included (1) middleware having to be configured, tested, and amended due to the changes brought about by a phased approach, and (2) a substantial workload being transferred from a nearby laboratory without provision of additional testing or staffing capacity. \n<\/p><p>For staff and personnel considerations, the following were identified: (1) after-hours authorization delay (by pathologists); (2) industrial action leading to delays; (3) paediatric and low-volume samples requiring manual processing, causing bottlenecks in the workflow; (4) staff constraints (in terms of insufficient staff to manage the benches); (5) additional training of staff being required for new procedures and processes for the platform testing changes implemented; (6) first-line manual sample preparation being needed to enable sample testing prior to full automation; and (7) training for the new platforms provided occurring on-site, while at the same time staff were also required to attend training off-site, leaving benches poorly staffed during training periods.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Cassim_AfricanJLabMed2020_9-2-b.jpg\" class=\"image wiki-link\" data-key=\"9964beabb3a442a59f9db1881f5b3333\"><img alt=\"Fig3 Cassim AfricanJLabMed2020 9-2-b.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/52\/Fig3_Cassim_AfricanJLabMed2020_9-2-b.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3.<\/b> Root cause analysis diagram developed in conjunction with the laboratory manager at a high-volume laboratory, Gauteng, South Africa, 2017<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>In this study, it was hypothesized that the application of appropriate corrective action, guided by an interactive TAT dashboard indicating the proportion of samples within stipulated TAT cutoffs and tail size (outliers), would result in improved performance.<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-1\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_10-0\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-10\">[10]<\/a><\/sup> This was based on the assumption that delivering an interactive TAT dashboard indicating outlier performance on its own would not result in improvement. Good laboratory management and response with appropriate corrective action is the key catalyst to deliver a sustained quality of reporting and ensure the continual TAT improvement of performance over time.<sup id=\"rdp-ebb-cite_ref-EzzelleGuide08_11-0\" class=\"reference\"><a href=\"#cite_note-EzzelleGuide08-11\">[11]<\/a><\/sup>\n<\/p><p>All laboratories typically adhere to a <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_management_system\" title=\"Quality management system\" class=\"wiki-link\" data-key=\"dfecf3cd6f18d4a5e9ac49ca360b447d\">quality management system<\/a> (QMS) that is used to assess laboratory quality from the pre-analytic phase through the testing and reviewing processes. A QMS is defined as a set of coordinated activities that direct and control a laboratory with regard to quality.<sup id=\"rdp-ebb-cite_ref-WHOLab11_12-0\" class=\"reference\"><a href=\"#cite_note-WHOLab11-12\">[12]<\/a><\/sup> All aspects of the laboratory operation, including the organizational structure, processes, and procedures need to be addressed by the QMS to assure quality.<sup id=\"rdp-ebb-cite_ref-WHOLab11_12-1\" class=\"reference\"><a href=\"#cite_note-WHOLab11-12\">[12]<\/a><\/sup> Laboratory quality can be defined in terms of accuracy, reliability, and timeliness (i.e., TAT).<sup id=\"rdp-ebb-cite_ref-WHOLab11_12-2\" class=\"reference\"><a href=\"#cite_note-WHOLab11-12\">[12]<\/a><\/sup> One of the key practices for <a href=\"https:\/\/www.limswiki.org\/index.php\/Continual_improvement_process\" title=\"Continual improvement process\" class=\"wiki-link\" data-key=\"fd7b54be3f6cdd0e8ed84d501486d668\">continual improvement<\/a> is the management review meeting, allowing the laboratory an opportunity to review annual performance as set out in the QMS. The management review cycle involves planning, implementing, checking, and acting on a quarterly basis to address shortfalls identified to affect continual improvement.<sup id=\"rdp-ebb-cite_ref-WHOLab11_12-3\" class=\"reference\"><a href=\"#cite_note-WHOLab11-12\">[12]<\/a><\/sup>\n<\/p><p>In this study, we reveal how the introduction of a TAT dashboard enabled senior management of the laboratory in question to assess TAT performance for a particular battery of tests that had not met the stipulated TAT cutoff (greater than 65% of results were outside of the stipulated TAT cutoff). Upon introduction of the dashboard, several areas of concern were immediately identified, including pre-analytical, analytical, and post-analytical factors. With respect to component TAT, assessing specifically the timeframes of \"registration to testing\" and \"testing to release of the report,\" TAT delays were attributable to delays of review during weeks one to 41, with further delays caused by testing (instrument) interruptions during weeks four and 48. The root cause analysis revealed several contributing factors, categorized as equipment and supplies; environmental; rules, policies, and procedures; and personnel or staffing issues. Specifically, the introduction of an auto-review rule process played an important role in improving TAT cutoff. A similarly placed high-volume core laboratory in Canada also reported that the implementation of a series of <a href=\"https:\/\/www.limswiki.org\/index.php\/Lean_laboratory\" title=\"Lean laboratory\" class=\"wiki-link\" data-key=\"6adad97006cb4eb4cac7dfcf767e3d5c\">lean approaches<\/a> in their busy laboratory, including automation and auto-review rules, proved effective for more efficiently managing substantial volumes of samples while meeting TAT cutoffs.<sup id=\"rdp-ebb-cite_ref-LouMulti17_13-0\" class=\"reference\"><a href=\"#cite_note-LouMulti17-13\">[13]<\/a><\/sup>\n<\/p><p>Several important lessons learned and documented by the study laboratory could serve as a template for outreach training to help other public sector laboratories achieve similar TAT performance improvements and establish the practices adopted at this site. Firstly, the importance of the need to collate and actively review real-time information about TAT\u2014including components of TAT, in ensuring overall timely reporting in laboratories\u2014was understood and confirmed. Secondly, the value of vertical audits was demonstrated. Vertical audits assisted in understanding what contributed to delayed TAT, and specific focus on outlier samples and vertical audits directed subsequent meaningful corrective action. In line with the requirement of ongoing improvement of service delivery, a weekly \"results for action\" statement was developed and found to be useful to deliver specific information at the sample or episode level. Such reports, while getting the attention of senior management, could be directed to relevant managers to highlight specific problematic areas and guide the focus of managers\u2019 attention to the investigation of true TAT outliers or exceptions. Such investigation (with solutions) of specifically identified problem areas could yield practical and advantageous outcomes, not only solving the issues at hand, but more widely having a positive impact on overall service delivery improvement. \n<\/p><p>The final lesson learned revolved around the importance of documenting and following through on corrective actions as part of the QMS. This ensures that corrective actions taken have consequences and are sustained. In the services review presented here, the week-by-week reminders of outlying TAT were a constant cue that solutions implemented had not been effective. Re-evaluation and re-assessment allowed for streamlined processes to be considered when initial corrective actions had failed. Also highlighted was the importance of conducting a root cause analysis using cause-and-effect diagrams, to tease out and understand all aspects of errors and any contributing factors that may lead to delays in TAT. It is also important to point out that although a corrective action may be resolved with a single intervention, more frequently, corrective action proves to be a multi-step process to identify possible solutions and alternatives. Once implemented, these corrective steps require consistent monitoring and evaluation for sustained impact. Here, the information provided by the dashboard offered objective evidence of identified issues that could be documented and presented to senior managers from month to month and at the annual management meeting; that, in due course, enabled corrective action planning and the facilitated, necessary mandates to affect better service.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Past_approaches_to_improving_turnaround_times\">Past approaches to improving turnaround times<\/span><\/h3>\n<p>Over the years, there have been multiple approaches to monitoring TAT reporting with the aim of improving TAT and, in turn, patient care. Approaches include identifying outliers, implementing Lean Sigma Six, and performing process mapping.<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-2\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BarakauskasProcess16_14-0\" class=\"reference\"><a href=\"#cite_note-BarakauskasProcess16-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HollandReduc05_15-0\" class=\"reference\"><a href=\"#cite_note-HollandReduc05-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrabanLean08_16-0\" class=\"reference\"><a href=\"#cite_note-GrabanLean08-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-StapletonImprov12_17-0\" class=\"reference\"><a href=\"#cite_note-StapletonImprov12-17\">[17]<\/a><\/sup> One of the earliest approaches described was the identification of TAT outliers.<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-3\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup> In 2005, Holland <i>et al.<\/i> reported that the average length of stay in the emergency department across 11 hospitals correlated significantly with the percentage of outliers.<sup id=\"rdp-ebb-cite_ref-HollandReduc05_15-1\" class=\"reference\"><a href=\"#cite_note-HollandReduc05-15\">[15]<\/a><\/sup> In 2007, Hawkins defined outliers as the TAT of samples that exceeded the institution\u2019s agreed TAT cutoffs.<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-4\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup> Outliers could also be defined as the tail size given the skewed TAT distribution. Therefore, the mean and confidence interval are not appropriate measures for assessing TAT performance. In 2018, Coetzee <i>et al.<\/i> used the 75th percentile and the percentage of samples within the cutoff TAT to identify outliers at the laboratory and test levels.<sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_10-1\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-10\">[10]<\/a><\/sup> The development of routine monitoring systems to identify laboratories with a long tail size enabled focused interventions to proactively resolve poor service delivery.<sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_10-2\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-10\">[10]<\/a><\/sup>\n<\/p><p>Many laboratories have implemented Lean Six Sigma approaches to improve TAT performance.<sup id=\"rdp-ebb-cite_ref-LouMulti17_13-1\" class=\"reference\"><a href=\"#cite_note-LouMulti17-13\">[13]<\/a><\/sup> \"Lean\" is defined as a continual improvement process consisting of technical tools and management methods.<sup id=\"rdp-ebb-cite_ref-GrabanLean08_16-1\" class=\"reference\"><a href=\"#cite_note-GrabanLean08-16\">[16]<\/a><\/sup> One of the aims of a lean approach is to find and eliminate waste.<sup id=\"rdp-ebb-cite_ref-GrabanLean08_16-2\" class=\"reference\"><a href=\"#cite_note-GrabanLean08-16\">[16]<\/a><\/sup> For example, waste may be introduced by the layout of a laboratory or by poor process design.<sup id=\"rdp-ebb-cite_ref-GrabanLean08_16-3\" class=\"reference\"><a href=\"#cite_note-GrabanLean08-16\">[16]<\/a><\/sup> Long distances between the receiving office and testing laboratory could also, for example, encourage staff to move samples in batches resulting in TAT delays.<sup id=\"rdp-ebb-cite_ref-GrabanLean08_16-4\" class=\"reference\"><a href=\"#cite_note-GrabanLean08-16\">[16]<\/a><\/sup> In 2008, Graban and Padgett reported that the introduction of lean approaches for troponin testing resulted in substantial TAT improvements and averted a proposed point-of-care testing implementation.<sup id=\"rdp-ebb-cite_ref-GrabanLean08_16-5\" class=\"reference\"><a href=\"#cite_note-GrabanLean08-16\">[16]<\/a><\/sup> A 2012 report by Stapleton and Sammond showed how a lean approach implemented over a three-year period in the laboratory resulted in TAT reductions for all emergency tests by implementing both workflow improvements and a dedicated emergency bench.<sup id=\"rdp-ebb-cite_ref-StapletonImprov12_17-1\" class=\"reference\"><a href=\"#cite_note-StapletonImprov12-17\">[17]<\/a><\/sup>\n<\/p><p>Another approach to improve TAT was reported by Barakauskas <i>et al.<\/i> in 2016, involving the use of LIS time stamps, direct observations, and discussions with staff to construct various value stream and process maps for immunosuppressant drug level testing.<sup id=\"rdp-ebb-cite_ref-BarakauskasProcess16_14-1\" class=\"reference\"><a href=\"#cite_note-BarakauskasProcess16-14\">[14]<\/a><\/sup> The value stream map identified process bottlenecks that were addressed.<sup id=\"rdp-ebb-cite_ref-BarakauskasProcess16_14-2\" class=\"reference\"><a href=\"#cite_note-BarakauskasProcess16-14\">[14]<\/a><\/sup> The process map was reported in columns to represent the major groups of personnel and locations from the health care worker to the reference testing laboratory, with the sequence of events and steps involved illustrated in a vertical direction.<sup id=\"rdp-ebb-cite_ref-BarakauskasProcess16_14-3\" class=\"reference\"><a href=\"#cite_note-BarakauskasProcess16-14\">[14]<\/a><\/sup> Bottlenecks were also identified in the process map to plan improvement initiatives, e.g., the use of emergency bags.<sup id=\"rdp-ebb-cite_ref-BarakauskasProcess16_14-4\" class=\"reference\"><a href=\"#cite_note-BarakauskasProcess16-14\">[14]<\/a><\/sup>\n<\/p><p>Ultimately, the aim of any of the approaches described above is to improve TAT performance and, thus, patient care. Although TAT is especially important for emergency tests that have very short TAT cutoffs, it is equally important to set cutoffs for other, less urgent tests to ensure that respective test results are received by attending physicians in a timely fashion to affect appropriate patient care, an important factor in assuring both the quality of care and the cost-effective use of hospital services.<sup id=\"rdp-ebb-cite_ref-WinkelmanHowFast97_18-0\" class=\"reference\"><a href=\"#cite_note-WinkelmanHowFast97-18\">[18]<\/a><\/sup> Aside from patient care, TAT delays also have the potential to waste valuable health resources caused by duplicate test requests, thereby increasing public health expenditure.<sup id=\"rdp-ebb-cite_ref-WinkelmanHowFast97_18-1\" class=\"reference\"><a href=\"#cite_note-WinkelmanHowFast97-18\">[18]<\/a><\/sup> In summary, the multiple approaches to improving TAT performance across all laboratory tests play an important role in improving the quality of patient care.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Application_to_African_contexts\">Application to African contexts<\/span><\/h3>\n<p>Figure 4 describes an approach that would make it possible for laboratories in low- and middle-income countries to collate the data required to develop the TAT reporting described in our study. In the first instance, laboratories would require a basic LIS that could generate weekly data extracts as described in Figure 4, using open-source database software such as Microsoft SQL Server (Express, Redmond, California, United States).<sup id=\"rdp-ebb-cite_ref-MSSQLServer_19-0\" class=\"reference\"><a href=\"#cite_note-MSSQLServer-19\">[19]<\/a><\/sup> Equally, <a href=\"https:\/\/www.limswiki.org\/index.php\/MySQL\" title=\"MySQL\" class=\"wiki-link\" data-key=\"35005451bfcd508bce47c58e72260128\">MySQL<\/a><sup id=\"rdp-ebb-cite_ref-MySQL_20-0\" class=\"reference\"><a href=\"#cite_note-MySQL-20\">[20]<\/a><\/sup>, Firebird<sup id=\"rdp-ebb-cite_ref-Firebird_21-0\" class=\"reference\"><a href=\"#cite_note-Firebird-21\">[21]<\/a><\/sup>, or <a href=\"https:\/\/www.limswiki.org\/index.php\/CUBRID\" title=\"CUBRID\" class=\"wiki-link\" data-key=\"0782bf9ee8c6aebfe9a037826975599a\">CUBRID<\/a><sup id=\"rdp-ebb-cite_ref-Cubrid_22-0\" class=\"reference\"><a href=\"#cite_note-Cubrid-22\">[22]<\/a><\/sup> could be deployed to generate the aggregate data described. Training is freely accessible via the internet for these software packages. There are multiple free online courses by providers via platforms such as edX, where one can learn how to both develop and query a structured query language (SQL) database.<sup id=\"rdp-ebb-cite_ref-edXQuerying_23-0\" class=\"reference\"><a href=\"#cite_note-edXQuerying-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-edXDevelopingArch_24-0\" class=\"reference\"><a href=\"#cite_note-edXDevelopingArch-24\">[24]<\/a><\/sup> Any of these software packages could be implemented on a local desktop or in a server environment, depending on the data volume, with very basic query tools using SQL commands, making it possible to develop dashboard tools using the step-by-step building blocks approach described in Figure 4.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Cassim_AfricanJLabMed2020_9-2-b.jpg\" class=\"image wiki-link\" data-key=\"492f50d7f289c7c4fbe15b8e9c6d6f09\"><img alt=\"Fig4 Cassim AfricanJLabMed2020 9-2-b.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/7e\/Fig4_Cassim_AfricanJLabMed2020_9-2-b.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 4.<\/b> Integration of laboratory information system specimen-level turnaround time data from multiple laboratories into a single, aggregated structured query language database for development of an interactive dashboard, Gauteng, South Africa, 2017<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Cross-country collaboration and sharing of resources could play an important role in securing already developed dashboard tools for other African countries. A multi-country approach could reduce overall costs and effort. For example, a single TAT dashboard could be developed for the Southern African Development Community to ensure accessibility and provide scalability. To secure the system and provide confidentiality, each country could have access to their own data using data access privileges. The benefit of this approach is that after the methods, systems, and dashboards are developed, it is easy to extend these developments to other countries with minimal additional cost. The only additional effort required at the country level would be to collate and share the data extracts with the umbrella organizers.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>In summary, this study demonstrated that an interactive TAT dashboard that reports appropriate TAT parameters, applied in the context of a QMS and coupled with proactive and diligent management, can accurately identify outliers and lead to appropriate corrective action and sustained timely laboratory reporting.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Limitations\">Limitations<\/span><\/h3>\n<p>Only LIS data were used for our study. Without a laboratory specimen tracking system, it is not possible to report end-to-end TAT. The implementation of an end-to-end tracking system from the time of venesection to delivery into a laboratory, additionally integrated into a TAT dashboard, could provide valuable supplementary date and time values to allow for an extended TAT efficiency review.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Lessons_from_the_field\">Lessons from the field<\/span><\/h3>\n<p>Our experiences in the field allow us provide a few lessons learned:\n<\/p>\n<ul><li> A weekly interactive TAT dashboard enables reporting of appropriate TAT paramaters and respective outcomes by confirming ongoing quality and timely reporting, as well as identifying outlying TAT that may require appropriate corrective action.<\/li><\/ul>\n<ul><li> TAT data can be collected at the laboratory, local network, or national level. A dashboard that includes aggregated and local level data, with a drill-down function, allows hierarchical review of the data, such that both higher-level managers and laboratory managers are able to view the same data, but at different levels appropriate for their respective level of responsibility.<\/li><\/ul>\n<ul><li> Continuously collating and analyzing the data and presenting TAT information in a user-friendly, visual dashboard format allows for immediate attention to be focused on outlying sites and areas.<\/li><\/ul>\n<ul><li> Visibility and transparency of TAT data and outcomes to all levels of management provide an incentive (with repeated peer or organization pressure, if consistently outside of TAT) to act on poor performance.<\/li><\/ul>\n<ul><li> A quality management system requires active input, monitoring, and appropriate action where needed. The presentation of information does not necessarily confer good performance or the meeting of TAT cutoffs. A dashboard, such as that presented here, is merely a tool. Proactive, consistent, and diligent review of TAT data presented in a dashboard is required to facilitate meaningful improvement and corrective action. An auto-review rule implemented for a specific test or battery of tests on the LIS has the potential to reduce TAT by acting to reduce the workload for senior staff through automatic review of predominantly normal results. With the auto-review implemented, senior staff effectively use their time and reserve the resultant review only for samples that fail to meet the auto-review rule, e.g., as with delta check failures.<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors thank area, business, and laboratory managers in Gauteng for their participation and input.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>D.K.G. supervised the study by providing leadership and oversight as the project leader. N.C., M.E.E.T. and L.M.C. designed the study, developed the methodology and conducted the research. M.E.E.T. extracted the data. N.C. conducted the data analysis. L.P. provided content for the cause and effect diagram and provided details on the interventions implemented. D.K.G. provided editorial comments and technical input. All authors reviewed the results and contributed to the manuscript development.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Disclaimer\">Disclaimer<\/span><\/h3>\n<p>The authors declare that the views expressed in the submitted article are their own and not the official position of any institution or funder.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare no conflict of interest. The authors further declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HawkinsLab07-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HawkinsLab07_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-HawkinsLab07_1-1\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-HawkinsLab07_1-2\">1.2<\/a><\/sup> <sup><a href=\"#cite_ref-HawkinsLab07_1-3\">1.3<\/a><\/sup> <sup><a href=\"#cite_ref-HawkinsLab07_1-4\">1.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hawkins, R.C. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2282400\" target=\"_blank\">\"Laboratory turnaround time\"<\/a>. <i>The Clinical Biochemist<\/i> <b>28<\/b> (4): 179\u201394. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2282400\/\" target=\"_blank\">PMC2282400<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18392122\" target=\"_blank\">18392122<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2282400\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2282400<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+turnaround+time&rft.jtitle=The+Clinical+Biochemist&rft.aulast=Hawkins%2C+R.C.&rft.au=Hawkins%2C+R.C.&rft.date=2007&rft.volume=28&rft.issue=4&rft.pages=179%E2%80%9394&rft_id=info:pmc\/PMC2282400&rft_id=info:pmid\/18392122&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2282400&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NHLSAnn1819-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NHLSAnn1819_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-NHLSAnn1819_2-1\">2.1<\/a><\/sup> <sup><a href=\"#cite_ref-NHLSAnn1819_2-2\">2.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">National Health Laboratory Service (2018) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/nationalgovernment.co.za\/entity_annual\/1714\/2018-national-health-laboratory-service-(nhls)-annual-report.pdf\" target=\"_blank\"><i>National Health Laboratory Service Annual Report 2017\/18<\/i><\/a>. National Health Laboratory Service. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780621457018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/nationalgovernment.co.za\/entity_annual\/1714\/2018-national-health-laboratory-service-(nhls)-annual-report.pdf\" target=\"_blank\">https:\/\/nationalgovernment.co.za\/entity_annual\/1714\/2018-national-health-laboratory-service-(nhls)-annual-report.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=National+Health+Laboratory+Service+Annual+Report+2017%2F18&rft.aulast=National+Health+Laboratory+Service&rft.au=National+Health+Laboratory+Service&rft.date=2018&rft.pub=National+Health+Laboratory+Service&rft.isbn=9780621457018&rft_id=https%3A%2F%2Fnationalgovernment.co.za%2Fentity_annual%2F1714%2F2018-national-health-laboratory-service-%28nhls%29-annual-report.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoetzeeTheImport18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CoetzeeTheImport18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coetzee, L.M.; Cassim, N.; Tepper, M.E.E. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchgate.net\/publication\/329610644\" target=\"_blank\">\"The importance of reporting individual weekly laboratory turn-around-time (TAT) to identify outliers and underperformance masked during global annual TAT review\"<\/a>. <i>Proceedings of the African Society for Laboratory Medicine Conference 2018<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.researchgate.net\/publication\/329610644\" target=\"_blank\">https:\/\/www.researchgate.net\/publication\/329610644<\/a><\/span>. \"Poster ID: PS-2.3b-070\"<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+importance+of+reporting+individual+weekly+laboratory+turn-around-time+%28TAT%29+to+identify+outliers+and+underperformance+masked+during+global+annual+TAT+review&rft.jtitle=Proceedings+of+the+African+Society+for+Laboratory+Medicine+Conference+2018&rft.aulast=Coetzee%2C+L.M.%3B+Cassim%2C+N.%3B+Tepper%2C+M.E.E.+et+al.&rft.au=Coetzee%2C+L.M.%3B+Cassim%2C+N.%3B+Tepper%2C+M.E.E.+et+al.&rft.date=2018&rft_id=https%3A%2F%2Fwww.researchgate.net%2Fpublication%2F329610644&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Microsoft-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Microsoft_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-Microsoft_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-Microsoft_4-2\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.microsoft.com\/en-za\/microsoft-365\/products-apps-services\" target=\"_blank\">\"Apps and services\"<\/a>. Microsoft<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.microsoft.com\/en-za\/microsoft-365\/products-apps-services\" target=\"_blank\">https:\/\/www.microsoft.com\/en-za\/microsoft-365\/products-apps-services<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 March 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Apps+and+services&rft.atitle=&rft.pub=Microsoft&rft_id=https%3A%2F%2Fwww.microsoft.com%2Fen-za%2Fmicrosoft-365%2Fproducts-apps-services&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MicroStratDesk-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MicroStratDesk_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.microstrategy.com\/en\/get-started\/desktop\" target=\"_blank\">\"Download MicroStrategy Desktop\"<\/a>. MicroStrategy<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.microstrategy.com\/en\/get-started\/desktop\" target=\"_blank\">https:\/\/www.microstrategy.com\/en\/get-started\/desktop<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Download+MicroStrategy+Desktop&rft.atitle=&rft.pub=MicroStrategy&rft_id=https%3A%2F%2Fwww.microstrategy.com%2Fen%2Fget-started%2Fdesktop&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MicrosoftDrill-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MicrosoftDrill_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Microsoft. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/docs.microsoft.com\/en-us\/power-bi\/consumer\/end-user-drill\" target=\"_blank\">\"Drill mode in a visual in Power BI\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/docs.microsoft.com\/en-us\/power-bi\/consumer\/end-user-drill\" target=\"_blank\">https:\/\/docs.microsoft.com\/en-us\/power-bi\/consumer\/end-user-drill<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Drill+mode+in+a+visual+in+Power+BI&rft.atitle=&rft.aulast=Microsoft&rft.au=Microsoft&rft_id=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpower-bi%2Fconsumer%2Fend-user-drill&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CassimFac1_20-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CassimFac1_20_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cassim, N.; Tepper, M.E.; Coetzee, L.M.; Glencross, D.K. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7203318\" target=\"_blank\">\"Timely delivery of laboratory efficiency information, Part I: Developing an interactive turnaround time dashboard at a high-volume laboratory\"<\/a>. <i>African Journal of Laboratory Medicine<\/i> <b>9<\/b> (2): a947. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v9i2.947\" target=\"_blank\">10.4102\/ajlm.v9i2.947<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7203318\/\" target=\"_blank\">PMC7203318<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32391244\" target=\"_blank\">32391244<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7203318\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7203318<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Timely+delivery+of+laboratory+efficiency+information%2C+Part+I%3A+Developing+an+interactive+turnaround+time+dashboard+at+a+high-volume+laboratory&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Cassim%2C+N.%3B+Tepper%2C+M.E.%3B+Coetzee%2C+L.M.%3B+Glencross%2C+D.K.&rft.au=Cassim%2C+N.%3B+Tepper%2C+M.E.%3B+Coetzee%2C+L.M.%3B+Glencross%2C+D.K.&rft.date=2020&rft.volume=9&rft.issue=2&rft.pages=a947&rft_id=info:doi\/10.4102%2Fajlm.v9i2.947&rft_id=info:pmc\/PMC7203318&rft_id=info:pmid\/32391244&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7203318&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhanRoot14-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KhanRoot14_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-KhanRoot14_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khan, K.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4064846\" target=\"_blank\">\"Root Cause Analysis (RCA) of Prolonged Laboratory Turnaround Time in a Tertiary Care Set Up\"<\/a>. <i>Journal of Clinical and Diagnostic Research<\/i> <b>8<\/b> (4): FC05\u2013FC08. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7860%2FJCDR%2F2014%2F7269.4255\" target=\"_blank\">10.7860\/JCDR\/2014\/7269.4255<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4064846\/\" target=\"_blank\">PMC4064846<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24959450\" target=\"_blank\">24959450<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4064846\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4064846<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Root+Cause+Analysis+%28RCA%29+of+Prolonged+Laboratory+Turnaround+Time+in+a+Tertiary+Care+Set+Up&rft.jtitle=Journal+of+Clinical+and+Diagnostic+Research&rft.aulast=Khan%2C+K.&rft.au=Khan%2C+K.&rft.volume=8&rft.issue=4&rft.pages=FC05%E2%80%93FC08&rft_id=info:doi\/10.7860%2FJCDR%2F2014%2F7269.4255&rft_id=info:pmc\/PMC4064846&rft_id=info:pmid\/24959450&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4064846&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IshikawaGuide76-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IshikawaGuide76_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ishikawa, K. (1976). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/archive.org\/details\/guidetoqualityco00ishi\/page\/n7\/mode\/2up\" target=\"_blank\"><i>Guide to Quality Control<\/i><\/a> (2nd Revised ed.). Asian Productivity Organization. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9283310365<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/archive.org\/details\/guidetoqualityco00ishi\/page\/n7\/mode\/2up\" target=\"_blank\">https:\/\/archive.org\/details\/guidetoqualityco00ishi\/page\/n7\/mode\/2up<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Guide+to+Quality+Control&rft.aulast=Ishikawa%2C+K.&rft.au=Ishikawa%2C+K.&rft.date=1976&rft.edition=2nd+Revised&rft.pub=Asian+Productivity+Organization&rft.isbn=9283310365&rft_id=https%3A%2F%2Farchive.org%2Fdetails%2Fguidetoqualityco00ishi%2Fpage%2Fn7%2Fmode%2F2up&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoetzeeUsing18-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CoetzeeUsing18_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_10-1\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_10-2\">10.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coetzee, L.-M.; Cassim, N.; Glencross, D.K. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6111574\" target=\"_blank\">\"Using laboratory data to categorise CD4 laboratory turn-around-time performance across a national programme\"<\/a>. <i>African Journal of Laboratory Medicine<\/i> <b>7<\/b> (1): 665. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v7i1.665\" target=\"_blank\">10.4102\/ajlm.v7i1.665<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6111574\/\" target=\"_blank\">PMC6111574<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30167387\" target=\"_blank\">30167387<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6111574\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6111574<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+laboratory+data+to+categorise+CD4+laboratory+turn-around-time+performance+across+a+national+programme&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Coetzee%2C+L.-M.%3B+Cassim%2C+N.%3B+Glencross%2C+D.K.&rft.au=Coetzee%2C+L.-M.%3B+Cassim%2C+N.%3B+Glencross%2C+D.K.&rft.date=2018&rft.volume=7&rft.issue=1&rft.pages=665&rft_id=info:doi\/10.4102%2Fajlm.v7i1.665&rft_id=info:pmc\/PMC6111574&rft_id=info:pmid\/30167387&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6111574&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EzzelleGuide08-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EzzelleGuide08_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ezzelle, J.; Rodriguez-Chavez, I.R.; Darden, J.M. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2213906\" target=\"_blank\">\"Guidelines on good clinical laboratory practice: bridging operations between research and clinical research laboratories\"<\/a>. <i>Journal of Pharmaceutical and Biomedical Analysis<\/i> <b>46<\/b> (1): 18\u201329. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpba.2007.10.010\" target=\"_blank\">10.1016\/j.jpba.2007.10.010<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2213906\/\" target=\"_blank\">PMC2213906<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18037599\" target=\"_blank\">18037599<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2213906\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2213906<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Guidelines+on+good+clinical+laboratory+practice%3A+bridging+operations+between+research+and+clinical+research+laboratories&rft.jtitle=Journal+of+Pharmaceutical+and+Biomedical+Analysis&rft.aulast=Ezzelle%2C+J.%3B+Rodriguez-Chavez%2C+I.R.%3B+Darden%2C+J.M.+et+al.&rft.au=Ezzelle%2C+J.%3B+Rodriguez-Chavez%2C+I.R.%3B+Darden%2C+J.M.+et+al.&rft.date=2008&rft.volume=46&rft.issue=1&rft.pages=18%E2%80%9329&rft_id=info:doi\/10.1016%2Fj.jpba.2007.10.010&rft_id=info:pmc\/PMC2213906&rft_id=info:pmid\/18037599&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2213906&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WHOLab11-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WHOLab11_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-WHOLab11_12-1\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-WHOLab11_12-2\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-WHOLab11_12-3\">12.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">World Health Organization (2011) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.who.int\/ihr\/publications\/lqms_en.pdf\" target=\"_blank\"><i>Laboratory Quality Management System Handbook<\/i><\/a>. World Health Organization. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9789241548274<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.who.int\/ihr\/publications\/lqms_en.pdf\" target=\"_blank\">https:\/\/www.who.int\/ihr\/publications\/lqms_en.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Laboratory+Quality+Management+System+Handbook&rft.aulast=World+Health+Organization&rft.au=World+Health+Organization&rft.date=2011&rft.pub=World+Health+Organization&rft.isbn=9789241548274&rft_id=https%3A%2F%2Fwww.who.int%2Fihr%2Fpublications%2Flqms_en.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LouMulti17-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LouMulti17_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-LouMulti17_13-1\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lou, A.H.; Elnenaei, M.O.; Sadek, I. et al. (2017). \"Multiple pre- and post-analytical lean approaches to the improvement of the laboratory turnaround time in a large core laboratory\". <i>Clinical Biochemistry<\/i> <b>50<\/b> (15): 864\u201369. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.clinbiochem.2017.04.019\" target=\"_blank\">10.1016\/j.clinbiochem.2017.04.019<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28457964\" target=\"_blank\">28457964<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multiple+pre-+and+post-analytical+lean+approaches+to+the+improvement+of+the+laboratory+turnaround+time+in+a+large+core+laboratory&rft.jtitle=Clinical+Biochemistry&rft.aulast=Lou%2C+A.H.%3B+Elnenaei%2C+M.O.%3B+Sadek%2C+I.+et+al.&rft.au=Lou%2C+A.H.%3B+Elnenaei%2C+M.O.%3B+Sadek%2C+I.+et+al.&rft.date=2017&rft.volume=50&rft.issue=15&rft.pages=864%E2%80%9369&rft_id=info:doi\/10.1016%2Fj.clinbiochem.2017.04.019&rft_id=info:pmid\/28457964&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarakauskasProcess16-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BarakauskasProcess16_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-BarakauskasProcess16_14-1\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-BarakauskasProcess16_14-2\">14.2<\/a><\/sup> <sup><a href=\"#cite_ref-BarakauskasProcess16_14-3\">14.3<\/a><\/sup> <sup><a href=\"#cite_ref-BarakauskasProcess16_14-4\">14.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barakauskas, V.E.; Bradshaw, T.A.; Smith, L.D. et al. (2016). \"Process Optimization to Improve Immunosuppressant Drug Testing Turnaround Time\". <i>American Journal of Clinical Pathology<\/i> <b>146<\/b> (2): 182\u201390. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fajcp%2Faqw087\" target=\"_blank\">10.1093\/ajcp\/aqw087<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27453440\" target=\"_blank\">27453440<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Process+Optimization+to+Improve+Immunosuppressant+Drug+Testing+Turnaround+Time&rft.jtitle=American+Journal+of+Clinical+Pathology&rft.aulast=Barakauskas%2C+V.E.%3B+Bradshaw%2C+T.A.%3B+Smith%2C+L.D.+et+al.&rft.au=Barakauskas%2C+V.E.%3B+Bradshaw%2C+T.A.%3B+Smith%2C+L.D.+et+al.&rft.date=2016&rft.volume=146&rft.issue=2&rft.pages=182%E2%80%9390&rft_id=info:doi\/10.1093%2Fajcp%2Faqw087&rft_id=info:pmid\/27453440&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HollandReduc05-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HollandReduc05_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-HollandReduc05_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Holland, L.L.; Smith, L.L.; Blick, K.E. (2005). \"Reducing laboratory turnaround time outliers can reduce emergency department patient length of stay: An 11-hospital study\". <i>American Journal of Clinical Pathology<\/i> <b>124<\/b> (5): 672\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1309%2FE9QP-VQ6G-2FBV-MJ3B\" target=\"_blank\">10.1309\/E9QP-VQ6G-2FBV-MJ3B<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16203280\" target=\"_blank\">16203280<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reducing+laboratory+turnaround+time+outliers+can+reduce+emergency+department+patient+length+of+stay%3A+An+11-hospital+study&rft.jtitle=American+Journal+of+Clinical+Pathology&rft.aulast=Holland%2C+L.L.%3B+Smith%2C+L.L.%3B+Blick%2C+K.E.&rft.au=Holland%2C+L.L.%3B+Smith%2C+L.L.%3B+Blick%2C+K.E.&rft.date=2005&rft.volume=124&rft.issue=5&rft.pages=672%E2%80%934&rft_id=info:doi\/10.1309%2FE9QP-VQ6G-2FBV-MJ3B&rft_id=info:pmid\/16203280&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GrabanLean08-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GrabanLean08_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-GrabanLean08_16-1\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-GrabanLean08_16-2\">16.2<\/a><\/sup> <sup><a href=\"#cite_ref-GrabanLean08_16-3\">16.3<\/a><\/sup> <sup><a href=\"#cite_ref-GrabanLean08_16-4\">16.4<\/a><\/sup> <sup><a href=\"#cite_ref-GrabanLean08_16-5\">16.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Graban, M.; Padgett, S. (2008). \"Lean Laboratories: Competing with Methods From Toyota\". <i>Laboratory Medicine<\/i> <b>39<\/b> (11): 645\u201348. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1309%2FLMX0LEMR7R0USKUM\" target=\"_blank\">10.1309\/LMX0LEMR7R0USKUM<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Lean+Laboratories%3A+Competing+with+Methods+From+Toyota&rft.jtitle=Laboratory+Medicine&rft.aulast=Graban%2C+M.%3B+Padgett%2C+S.&rft.au=Graban%2C+M.%3B+Padgett%2C+S.&rft.date=2008&rft.volume=39&rft.issue=11&rft.pages=645%E2%80%9348&rft_id=info:doi\/10.1309%2FLMX0LEMR7R0USKUM&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StapletonImprov12-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-StapletonImprov12_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-StapletonImprov12_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stapleton, K.; Sammond, D. (2012). \"Improved Laboratory Stat Test Turnaround Time Using Lean Six Sigma\". <i>American Journal of Clinical Pathology<\/i> <b>138<\/b> (Suppl. 2): A184. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fajcp%2F138.suppl2.36\" target=\"_blank\">10.1093\/ajcp\/138.suppl2.36<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improved+Laboratory+Stat+Test+Turnaround+Time+Using+Lean+Six+Sigma&rft.jtitle=American+Journal+of+Clinical+Pathology&rft.aulast=Stapleton%2C+K.%3B+Sammond%2C+D.&rft.au=Stapleton%2C+K.%3B+Sammond%2C+D.&rft.date=2012&rft.volume=138&rft.issue=Suppl.+2&rft.pages=A184&rft_id=info:doi\/10.1093%2Fajcp%2F138.suppl2.36&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WinkelmanHowFast97-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WinkelmanHowFast97_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-WinkelmanHowFast97_18-1\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Winkelman, J.W.; Tanasijevic, M.J.; Wybenga, D.R. et al. (1997). \"How fast is fast enough for clinical laboratory turnaround time? Measurement of the interval between result entry and inquiries for reports\". <i>American Journal of Clinical Pathology<\/i> <b>108<\/b> (4): 400\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fajcp%2F108.4.400\" target=\"_blank\">10.1093\/ajcp\/108.4.400<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/9322592\" target=\"_blank\">9322592<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+fast+is+fast+enough+for+clinical+laboratory+turnaround+time%3F+Measurement+of+the+interval+between+result+entry+and+inquiries+for+reports&rft.jtitle=American+Journal+of+Clinical+Pathology&rft.aulast=Winkelman%2C+J.W.%3B+Tanasijevic%2C+M.J.%3B+Wybenga%2C+D.R.+et+al.&rft.au=Winkelman%2C+J.W.%3B+Tanasijevic%2C+M.J.%3B+Wybenga%2C+D.R.+et+al.&rft.date=1997&rft.volume=108&rft.issue=4&rft.pages=400%E2%80%935&rft_id=info:doi\/10.1093%2Fajcp%2F108.4.400&rft_id=info:pmid\/9322592&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MSSQLServer-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MSSQLServer_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Microsoft. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/sql-server-2019\" target=\"_blank\">\"SQL Server 2019\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/sql-server-2019\" target=\"_blank\">https:\/\/www.microsoft.com\/en-us\/sql-server\/sql-server-2019<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 April 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SQL+Server+2019&rft.atitle=&rft.aulast=Microsoft&rft.au=Microsoft&rft_id=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fsql-server%2Fsql-server-2019&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MySQL-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MySQL_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Oracle Corporation. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.mysql.com\/products\/community\/\" target=\"_blank\">\"MySQL Community Edition\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.mysql.com\/products\/community\/\" target=\"_blank\">https:\/\/www.mysql.com\/products\/community\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 February 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MySQL+Community+Edition&rft.atitle=&rft.aulast=Oracle+Corporation&rft.au=Oracle+Corporation&rft_id=https%3A%2F%2Fwww.mysql.com%2Fproducts%2Fcommunity%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Firebird-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Firebird_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Firebird Project. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/firebirdsql.org\/en\/firebird-rdbms\/\" target=\"_blank\">\"Firebird RDBMS\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/firebirdsql.org\/en\/firebird-rdbms\/\" target=\"_blank\">https:\/\/firebirdsql.org\/en\/firebird-rdbms\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 April 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Firebird+RDBMS&rft.atitle=&rft.aulast=Firebird+Project&rft.au=Firebird+Project&rft_id=https%3A%2F%2Ffirebirdsql.org%2Fen%2Ffirebird-rdbms%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Cubrid-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Cubrid_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">CUBRID Foundation. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cubrid.org\/\" target=\"_blank\">\"CUBRID\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cubrid.org\/\" target=\"_blank\">https:\/\/www.cubrid.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 April 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CUBRID&rft.atitle=&rft.aulast=CUBRID+Foundation&rft.au=CUBRID+Foundation&rft_id=https%3A%2F%2Fwww.cubrid.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-edXQuerying-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-edXQuerying_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Microsoft. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.edx.org\/course\/querying-data-with-transact-sql-2\" target=\"_blank\">\"Querying Data with Transact-SQL\"<\/a>. <i>edX<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.edx.org\/course\/querying-data-with-transact-sql-2\" target=\"_blank\">https:\/\/www.edx.org\/course\/querying-data-with-transact-sql-2<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 April 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Querying+Data+with+Transact-SQL&rft.atitle=edX&rft.aulast=Microsoft&rft.au=Microsoft&rft_id=https%3A%2F%2Fwww.edx.org%2Fcourse%2Fquerying-data-with-transact-sql-2&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-edXDevelopingArch-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-edXDevelopingArch_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Microsoft. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20200502074237\/http:\/\/www.edx.org\/course\/developing-sql-databases\" target=\"_blank\">\"Developing SQL Databases\"<\/a>. <i>edX<\/i>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.edx.org\/course\/developing-sql-databases\" target=\"_blank\">the original<\/a> on 20 May 2020<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20200502074237\/http:\/\/www.edx.org\/course\/developing-sql-databases\" target=\"_blank\">https:\/\/web.archive.org\/web\/20200502074237\/http:\/\/www.edx.org\/course\/developing-sql-databases<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 April 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Developing+SQL+Databases&rft.atitle=edX&rft.aulast=Microsoft&rft.au=Microsoft&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20200502074237%2Fhttp%3A%2F%2Fwww.edx.org%2Fcourse%2Fdeveloping-sql-databases&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The original citation number two (2017-2018 NHLS Annual Report) was dead; an alternately hosted version was found and used for this version. The original cites Wikipedia concerning the Ishikawa diagram; as using Wikipedia as a citation is generally frowned upon, a substitute citation for Ishikawa's original <i>Guide to Quality Control<\/i> was used for this version. The original article accidentally mixed up citations 19 and 20; this was corrected for this version. The citation URL for the SQL Databases edX course is dead; an archived version of the URL is used for this version. The original article used an image for lessons learned in the field; this version skips the image and replaces it with the image text.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194115\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.568 seconds\nReal time usage: 1.997 seconds\nPreprocessor visited node count: 19050\/1000000\nPreprocessor generated node count: 36869\/1000000\nPost\u2010expand include size: 142906\/2097152 bytes\nTemplate argument size: 47328\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 411.536 1 - -total\n 80.80% 332.507 1 - Template:Reflist\n 66.71% 274.529 24 - Template:Citation\/core\n 40.03% 164.741 12 - Template:Cite_journal\n 21.72% 89.379 9 - Template:Cite_web\n 9.29% 38.224 1 - Template:Infobox_journal_article\n 8.86% 36.449 1 - Template:Infobox\n 8.76% 36.054 3 - Template:Cite_book\n 6.08% 25.034 27 - Template:Citation\/identifier\n 6.05% 24.916 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12302-0!*!0!!en!5!* and timestamp 20210429194113 and revision id 41312\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory\">https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","2897b25895d56e2e3f8922c52ab067ea_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ee\/Fig1_Cassim_AfricanJLabMed2020_9-2-b.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/fb\/Fig2_Cassim_AfricanJLabMed2020_9-2-b.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/52\/Fig3_Cassim_AfricanJLabMed2020_9-2-b.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/7e\/Fig4_Cassim_AfricanJLabMed2020_9-2-b.jpg"],"2897b25895d56e2e3f8922c52ab067ea_timestamp":1619725273,"d11707b0ec9b9843d1f6332c3f2b2930_type":"article","d11707b0ec9b9843d1f6332c3f2b2930_title":"Timely delivery of laboratory efficiency information, Part I: Developing an interactive turnaround time dashboard at a high-volume laboratory (Cassim et al. 2020)","d11707b0ec9b9843d1f6332c3f2b2930_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory","d11707b0ec9b9843d1f6332c3f2b2930_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Timely delivery of laboratory efficiency information, Part I: Developing an interactive turnaround time dashboard at a high-volume laboratory\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nTimely delivery of laboratory efficiency information, Part I: Developing an interactive turnaround time dashboard at a high-volume laboratoryJournal\n \nAfrican Journal of Laboratory MedicineAuthor(s)\n \nCassim, Naseem; Tepper, Manfred E.; Coetzee, Lindi M.; Glencross, Deborah K.Author affiliation(s)\n \nNational Health Laboratory Service, University of the WitwatersrandPrimary contact\n \nEmail: naseem dot cassim at wits dot ac dot zaYear published\n \n2020Volume and issue\n \n9(2)Article #\n \na947DOI\n \n10.4102\/ajlm.v9i2.947ISSN\n \n2225-2010Distribution license\n \nCreative Commons Attribution 4.0 International LicenseWebsite\n \nhttps:\/\/ajlmonline.org\/index.php\/ajlm\/article\/view\/947\/1482Download\n \nhttps:\/\/ajlmonline.org\/index.php\/ajlm\/article\/download\/947\/1479 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methods \n\n3.1 Ethical considerations \n3.2 Study design \n3.3 Steps to developing a turnaround time dashboard \n3.4 Sample population and turnaround time definition \n3.5 Test basket development and inclusion and exclusion criteria \n3.6 Data extraction \n3.7 Criteria for an effective dashboard \n3.8 Data analysis and visual dashboard display \n\n\n4 Results \n5 Discussion \n6 Conclusion \n\n6.1 Limitations \n6.2 Lessons from the field \n\n\n7 Acknowledgements \n\n7.1 Author contributions \n7.2 Disclaimer \n7.3 Competing interests \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nBackground: Mean turnaround time (TAT) reporting for testing laboratories in a national network is typically static and not immediately available for meaningful corrective action and does not allow for test-by-test or site-by-site interrogation of individual laboratory performance.\nObjective: The aim of this study was to develop an easy-to-use, visual dashboard to report interactive graphical TAT data to provide a weekly snapshot of TAT efficiency.\nMethods: An interactive dashboard was developed by staff from the National Priority Programme and Central Data Warehouse of the National Health Laboratory Service in Johannesburg, South Africa, during 2018. Steps required to develop the dashboard were summarized in a flowchart. To illustrate the dashboard, one week of data from a busy laboratory for a specific set of tests was analyzed using annual performance plan TAT cutoffs. Data were extracted and prepared to deliver an aggregate extract, with statistical measures provided, including test volumes, global percentage of tests that were within TAT cutoffs, and percentile statistics.\nResults: Nine steps were used to develop the dashboard iteratively, with continuous feedback for each step. The data warehouse environment conformed and stored laboratory information system (LIS) data in two formats: (1) fact and (2) dimension. Queries were developed to generate an aggregate TAT data extract to create the dashboard. The dashboard successfully delivered weekly TAT reports.\nConclusion: Implementation of a TAT dashboard can successfully enable the delivery of near real-time information and provide a weekly snapshot of efficiency in the form of TAT performance to identify and quantitate bottlenecks in service delivery.\nKeywords: turnaround time, laboratory efficiency, interactive dashboard, indicators, performance assessment\n\nIntroduction \nTurnaround time (TAT) is an important performance indicator for a laboratory service. It refers to the time from first registration in a laboratory to a result released in the laboratory information system (LIS).[1] Historically, within the National Health Laboratory Service (NHLS) of South Africa, TAT reporting was provided in annual and quarterly static management reports generated by the LIS, TrakCare[2], which also provided ad hoc reporting for use at the laboratory level. These reports are printed to provide a snapshot of TAT reporting and are suited for staff working at the laboratory level. At a national level, the corporate data warehouse (CDW) of the NHLS collated global TAT data from over 266 testing laboratories based on predetermined, annual performance plan cutoffs. National TAT cutoffs are set by expert committees of different pathology disciplines, with final confirmation from senior management before implementation. These cutoffs are set with provisions for all levels of the service laboratory: from low-volume laboratories with limited test repertoires to high-volume testing laboratories with extensive test offerings, including specialized testing, such as viral load testing. However, a large percentage of NHLS laboratories have 24-hour service and have emergency units in the hospitals in which they are housed; such laboratories have locally stricter TAT cutoffs for emergency and other local tests than are reflected in the national cutoffs for all samples.\nHistorically, the NHLS CDW TAT reports generated were static and reported only the mean TAT. Turnaround time data have a positive skewness, that is, a long tail to the right, meaning that the mean will be greater than the median. This implies that TAT data reported previously[1][3], reporting the mean TAT, masks good performance while concealing poor efficiency. Further, neither the current LIS nor CDW reports enable detailed analysis of the information or drilling down to laboratory- or test-level data for additional information about TAT efficiencies. Data presented at the first conference of the African Society for Laboratory Medicine in Cape Town, South Africa, in 2012, reported daily laboratory test volumes and mean TAT for authorized results, stratified by individual laboratories[4], providing a snapshot of performance. This enabled review of CD4 laboratory efficiency for a national program and provided important insights into laboratory operations.\nA recent evaluation of TAT for HIV programs reported a methodology to further categorize laboratory TAT performance using three additional measures[3]: (1) median TAT, (2) 75th percentile TAT (tail size), and (3) percentage within cutoff TAT. These data were graphically presented using a scatter plot of percentage of samples within the TAT cutoff (x-axis) against the TAT 75th percentile (y-axis), categorized into four quadrants of performance to help identify the level of laboratory performance in a national program. This approach made it easier to identify both good performers and outliers in the same analysis.[3] The report was generated in Excel and included the raw monthly data, a scatter and bar graph, and a summary table of all laboratories per business region, the 75th percentile, the percentage within cutoff values, and TAT component 75th percentile values. This report was primarily distributed to managers of all HIV-related operational programs, that is, CD4, viral load and tuberculosis testing, and early infant diagnosis for review and intervention, and not shared across the network of testing laboratories.\nSenior management in Gauteng, South Africa expressed a need for a TAT monitoring tool that would enable them to better manage their laboratories and identify sites with poor TAT performance. Given the static nature of the historic TAT reporting in the organization, an interactive system that offered information to enable review of performance, including outliers (tail size assessment), while confirming that sites were meeting cutoffs, would be a useful tool to enable business and laboratory managers alike to monitor their efficiency via TAT performance in real time. The concepts already developed and in use as Excel reports for the HIV programs were the starting point for developing a reporting dashboard for use across multiple disciplines and tests done throughout the network of 241 testing laboratories of the NHLS in South Africa.\nThe aim of this study was to develop an easy-to-use information system, in a dashboard format. This would enable weekly reporting of TAT data as a snapshot of performance. To achieve this, a number of changes to current TAT reporting had to be addressed. These changes included (1) moving from program-specific, single-test TAT reporting previously used[3] to a specific set of high-volume tests, (2) adopting TAT measures reported by Coetzee et al.[3], (3) identifying dashboard software to use, and (4) identifying the target users.[3] The specific set of tests (or \"basket\" of commonly requested tests) should be representative across the primary pathology disciplines.\nThis article sets out to describe the process followed to develop the TAT dashboard\u2014using available software\u2014that could provide a weekly summary of national, business unit, and laboratory-level TAT performance for a basket of tests. For the purposes of this article, data from a single participating tertiary laboratory were used to illustrate the data distribution, as it represents an example of a testing facility that performed all tests reviewed in the prescribed national basket. This was done to show the respective levels of drilling functionality of the dashboard and to iterate the interactive properties, while demonstrating how the dashboard can be used to assess performance and identify outliers for intervention.\n\nMethods \nEthical considerations \nEthics clearance for this study was obtained from the University of the Witwatersrand (M1706108). Only anonymized laboratory data, containing no patient identifiers, were used for the study.\n\nStudy design \nA retrospective descriptive study design was used to analyze and report laboratory TAT data for a specific set of tests (Table 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Table 1. Sample of a turnaround time dashboard table that lists the outcomes for the basket of tests, South Africa, 2018\n\n\n\nSteps to developing a turnaround time dashboard \nThe various steps required to develop the dashboard are summarized in a flowchart (Figure 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1. Flowchart depicting all steps required to develop a turnaround time dashboard, South Africa, 2018\n\n\n\nSample population and turnaround time definition \nUsing convenience sampling, data were selected from among the tests performed at a single busy academic laboratory in Gauteng for one week during 2018. Aside from global TAT reporting, the dashboard should adopt the three TAT measures reported by Coetzee 'et al.[3] These include pre-analytical, analytical, and post-analytical components of TAT components, namely: (1) the time from first registration at the source laboratory to registration of the referral at the testing laboratory (LAB-to-LAB TAT), (2) time from registration at the testing laboratory to results being populated by the LIS interface (TESTING TAT), and (3) time from result population by the LIS interface to manual review and authorization by senior laboratory staff (REVIEW TAT).\n\nTest basket development and inclusion and exclusion criteria \nFocus group meetings were arranged with local area and business managers to define a test basket for the dashboard. The principles adopted were as follows: (1) measure a limited number of tests with a focus on the tests with the highest volumes of tests performed; (2) measure data for the indicator analyte (as a proxy) for specific panel tests (for example, the creatinine test was used as an indicator for assessment of urea and electrolyte test performance); (3) use the annual performance plan TAT cutoffs; and (4) deliver dashboard files via email (due to bandwidth constraints). All samples within this organization test basket were included in the example analysis and included the most commonly requested tests selected from hematology, coagulation, HIV-tuberculosis, and chemistry (Table 1). A mapping table was developed to identify the LIS test sets and items to be reported. For each test, the TAT cutoff was also stipulated. The mapping table was used to guide the data extract.\n\nData extraction \nFor the purposes of demonstrating how the data were manipulated to create the dashboard, data were extracted for the week of September 2\u20138, 2018 from the CDW from four data sources: (1) the Operational Data Store that contained the original LIS data (Figure 2), (2) the \"CDW fact\" that reported test volumes, (3) the test method dimension[5] (provides details on the test such as a unique identifier, discipline, test method code and name and national number from the CDW), and (4) TAT cutoff dimension (captures annual performance plan cutoffs) (Figure 1).[6] Using an outer join, data from these four data sources were prepared as a temporary detailed table. The first temporary table limited data to the test basket, adding the TAT cutoffs and provided information using the laboratory hierarchy (region, business unit, and laboratory). Because this table would be too large to use for the dashboard, and assuming email delivery of the final report, two additional steps were used to create a smaller aggregate dataset. The mean, standard deviation, 75th percentile, and percentage within TAT cutoff were added. All TAT data were reported in hours. The final temporary table was exported as a Microsoft Excel (Redmond, Washington, United States)[7] worksheet and imported into the MicroStrategy Desktop analytics tool (Providence, Virginia, United States).[8] After the data were imported, the respective dashboard sheets were developed to include relevant TAT information for all levels of management.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2. Visual representation of data preparation steps to transform and move raw turnaround time data, South Africa, 2018. Data were moved from the Operational Data Store (the laboratory information system) to a production server which consists of facts and dimensions. The production server structures the data by setting sets of data to specific target areas to facilitate final reporting in dashboard format.\n\n\n\nCriteria for an effective dashboard \nFor any dashboard to be effective, it needs to adhere to a number of key outputs: it should (1) be visually engaging and easy to view and understand the TAT data displayed; (2) enable dynamic drilling down from a bird\u2019s-eye view to a local perspective, that is, from national or provincial level down to the laboratory level per test; (3) provide a report on a weekly basis for a TAT snapshot view; and (4) highlight TAT outliers for laboratory managers to follow-up and direct corrective action. From a more technical perspective, the dashboard also had to include additional features that included (1) conditional formatting to highlight good, average, and poor performance; (2) provision of various reporting formats such as bubble charts, tables, and bar charts; (3) the ability to import data for a variety of formats; and (4) theability to send the weekly dashboard data file via email in a small file format (\u2264 6 MB).\n\nData analysis and visual dashboard display \nThe dashboard displays (sheets) developed were as follows: (1) a bubble chart reporting the percentage within TAT cutoffs and 75th percentiles, (2) a table (see Table 1) displaying the bubble chart data, and (3) the 75th percentile for each phase of the component TAT reported by the test method. A bubble chart dashboard sheet was created to include: (1) the 75th percentile TAT (y-axis), (2) the percentage within TAT cutoff (x-axis), (3) the test volumes (size by and color by), and (4) the test method (color by and break by). The region codes and laboratory names were added to the dashboard as filters (radio buttons and search box display styles). An 85% within TAT cutoff reference line was added to aid identification of specific tests and associated laboratories with TAT that were outside of the TAT cutoffs. The data used to generate the bubble chart were also reported as a table in a separate sheet. The table listed the test name, total number of tests, TAT cutoff, the percentage within TAT cutoff, and 75th percentile TAT for the basket of tests reported on.\nThe table uses \"stop highlighting\" to denote the different percentage within TAT cutoff as follows: (1) 85% or higher in green, (2) 75% \u2013 84% in orange, and (3) under 75% in red. Lower percentage within TAT cutoff and higher 75th percentiles indicate an increased risk that any given laboratory is not adequately delivering patient reports that will enable timely clinical intervention.\nA component TAT sheet was created as a clustered horizontal bar chart to display the component TAT with (1) test method name (y-axis) and (2) component 75th percentile TAT (LAB-LAB TAT, TESTING TAT and REVIEW TAT), differentiated by color. The testing laboratory name was added to enable refining and filtering data down to the laboratory level.\n\nResults \nThe successfully developed dashboard enabled delivery of weekly TAT data. Data from 45,599 reported samples for the week of September 2\u20138, 2018 were utilized to demonstrate the dashboard development described here. The 75th percentile and the percentage of tests within stipulated cutoff for each test in the basket were visualized on the dashboard landing screen. This allowed the user to view data by test at the national, provincial and laboratory levels to visually identify outlying tests. The dashboard contained three individual sheets: the bubble chart, the TAT table, and the component TAT sheets.\nFigure 3 shows typical weekly TAT data presentation outcomes as a bubble chart dashboard. In this example data set, only one test method, rapid plasma reagin (syphilis), failed to meet the 85% TAT cutoff and is reported as a small grey dot on the bubble chart dashboard. For this test, the reported percentage within cutoff was 83.8%, within the 75% and 84% category highlighted as orange in the prior Table 1. A cluster of tests in the bottom right reported 90% or higher within TAT cutoff with a 75th percentile TAT of eight hours or less. Only one test reported a percentage within TAT cutoff between 85% and 89%: total cholesterol (red dot). Higher test volumes were reported for the HIV viral load (n = 19,055) and creatinine (n = 8,857) tests.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 3. An example of the Microstrategy Desktop bubble dashboard chart used to report total turnaround time data for an example site\u2019s week\u2019s data, South Africa, 2018. The percentage within cutoff turnaround time is reported on the x-axis with the 75th percentile turnaround time on the y-axis. The bubble size indicates test volumes. Reference lines were added at 85% within stipulated turnaround time cutoff on the x-axis. Each test within the test basket is colour coded with the key provided on the right. Outlying tests are immediately visible.\n\n\n\nFor the TAT table, results for the bubble chart are summarized (see Table 1) per test. At the 75th percentile TAT, no test exceeded the cutoff TAT. A 100% within cutoff TAT was reported for three tests: activated partial thromboplastin time, full blood count, and platelet count. Similarly, six tests reported a percentage within cutoff TAT between 95% and 99%.\nThe dashboard also reports component TAT in hours (Figure 4), including (1) LAB-TO-LAB, (2) TESTING, and (3) REVIEW times, with the tail size in hours for the distribution of each component TAT. In any given laboratory, some samples tested are local (from the immediately adjacent hospital), while other samples are referred for testing from nearby hospitals where these tests are not available. As such, a zero LAB-TO-LAB component indicates that the samples were not referred but are samples collected and tested locally. For referred samples included in the example data set (see Figure 4, CD4 antiretrovirals, D-Dimer, and HIV viral load, among others), the LAB-TO-LAB component TAT 75th percentile represents the inter-laboratory referral time, ranging in this instance from 12 to 23 hours (Figure 4). In the testing phase, TAT ranged from 0.25 to 63 hours (where 63 hours represented a single test, the rapid plasma reagin, syphilis, that was regarded locally as an outlier; see Table 1 for detail). The 75th percentile review TAT was two hours or less across all tests.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 4. MicroStrategy Desktop dashboard bar chart used to report the component turnaround time data for an example site\u2019s for the week of September 2\u20138, 2018, South Africa. The components reported are LAB-LAB (inter-laboratory referral time), TESTING (time from registration to testing), and REVIEW (time from testing to review) turnaround time components.\n\n\n\nDiscussion \nAccess to information in an interactive dashboard format has previously enabled retrieval of health data for immediate clinical use in the NHLS in South Africa.[3] A similar approach has been applied and demonstrated in this work for TAT data. The dashboard described here provides an interactive, weekly snapshot of TAT performance, together with information about TAT distribution, tail size (outlier) assessment[1][3], to varying levels of laboratory managers across the NHLS, to enable timely intervention where poor service delivery is identified.\nThe dashboard is comprised of a few basic parameters that act together to provide information about TAT. Date stamping of samples in the LIS is a prerequisite to provide the basic information necessary to detail TAT linked to any given sample. Together with relevant sample identification datalogged, data is transferred to a central database for careful curation. Later, TAT data extraction is performed using standard data query tools. In the instance of a wider network of laboratories operating within the same organization, such as the South African NHLS, LIS data is stored using a decentralized architecture. Aggregate data, in the format described above, can then be collated and used to develop national TAT dashboards.\nThe dashboard described in this study simplifies presentation of complex data by enabling visualization of any given laboratory\u2019s efficiency. For the purpose of this study and to demonstrate the effectiveness and simple format of the dashboard developed, data from a single busy laboratory were used to illustrate the different outputs of the dashboard (graphs and table). The example data used here reveal how the dashboard can be used to identify tests that are not meeting national (or local) cutoff criteria. In the example presented, rapid plasma reagin (syphilis) testing was noted as an outlier, as it did not meet the organization-stipulated 85% within cutoff TAT. The summary table (example shown in Table 1) also provides a spreadsheet format table of the relevant tests either meeting, or failing to meet, the national cutoff criteria. The additional information on TAT component analysis further assists management to identify those areas of laboratory testing, within the respective pre-analytical, analytical, and post-analytical components, that may need investigation for improvement.\nThe dashboard was successfully rolled out to all NHLS testing laboratories; weekly data are currently received by these laboratories for review. The dashboard development included a drill-down function into the performance of a particular test to see results by testing laboratory, or business unit. The addition of tail size measures[1][3] has also enabled managers to identify less efficient areas of their laboratory services with outlying performance, seen in the example case described in this article (rapid plasma reagin \u2013 syphilis), which would have been otherwise missed using conventional reporting alone (using mean TAT reporting). In addition, to enable practical sample-by-sample audit, individual samples that did not meet cutoff criteria were identified for follow-up in an additional summary table sheet (added at the request of laboratory managers to enable better intervention; not shown). Use of the dashboard has also led to laboratory process changes with improved individual component TAT. For example, post-analytical TAT improvements included implementation of an \"auto review\" feature.[9][10] With respect to analytical delays identified, testing delays could be correlated with instrument breakdown logs from the laboratories or instrument suppliers to identify reasons for prolonged testing TAT.[9] The impact of dashboard usage on improving TAT is described in detail in the companion article in this issue.[9]\nRisk management teaches that not all errors can be predicted.[11] However, it is only through active review of quality processes that delays, errors, and problems can be detected earlier to enable corrective action. Thus, critical to managing risk is the continuous and ongoing evaluation and assessment of procedures and processes to ensure that the same errors are not repeated. Here, human capital is key to the sustainability and success of any dashboard implementation. Noble et al.[11] reported that only the persistence and interest of laboratory personnel to maintain quality can ensure smooth and rapid progress of error detection (and correction back to quality) that is fast and sustainable. \nOne of the fundamental lessons learned from the development of the dashboard described here is that providing tools to assess TAT performance does not in itself imply corrective action or improvement. The dashboard is merely a tool that enables managers to effectively and efficiently ensure procedural excellence. Nkengasong and Birx[12] also suggest that in order for innovation to be adopted, and sustainable innovation and performance enablers should both energize and incentivize laboratories across four pillars: implementation, measurement, reward, and improvement. A culture of diligence and willingness on the part of managers to meaningfully use information provided in the dashboard is thus important to enable making consequential changes at the laboratory level. Political will and strong senior leadership are also needed to make systems, such as those introduced with the dashboard described here, both functional and sustainable.[12] This can be done by appropriately recognizing and rewarding laboratories and personnel who use the tools provided.\nPre-analytical errors should not be underestimated, as they can increase both testing errors and TAT.[12] In the example laboratory performance reported here, all four referred test TAT outcomes were compromised due to pre-analytical delays. Documenting these delays and acting to reduce pre-analytical time, including travel time, and time spent in receiving centers prior to sample registration, can be used to streamline services.\nAnother outcome reported by managers using the dashboard was that the information could be documented week-by-week to provide objective evidence to document and motivate for additional resources required to achieve TAT cutoffs, for example including additional sample collection schedules, increasing testing capacity, and providing motivation for auto-review and authorization modules.[9]\nIt is important in the context of a resource-poor setting to highlight that the dashboard described here was developed without specific funding, relying only on the collaborative effort of NHLS staff (the authors) with data management or MicroStrategy skills. Data is routinely transferred from the LIS to the CDW, where is it collated and carefully curated for downstream research and operational needs. Initial formats were undertaken using CDW-extracted data analyzed in MS Excel to create simple charts plotting 75th percentile and median TAT, by laboratory, for annualized or quarterly aggregated TAT data. Thereafter, analyses were extended to create week-by-week practical and usable worksheets so that individual laboratories could view current data. Using MicroStrategy, a freely available software program, a dashboard was developed to enable automatic presentation of the data in a visible interactive format (with the snapshot aggregate data file emailed to users weekly) to facilitate automated more immediate access to current TAT data. Future planning includes providing live data in the dashboard, facilitated by extending local bandwidth capacity and immediate real-time analysis of data within the CDW itself.\n\nConclusion \nThis article outlines the database management and methods used for the development of a dashboard that enables presentation of weekly TAT data to relevant business and laboratory managers, as part of the overall quality management portfolio of the organization. This novel approach ensures the delivery of quality, timely pathology reporting by the South African NHLS and, ultimately, better patient care. Training on the use of the dashboard is essential to ensure that users are competent. Users need to both understand the principles applied in the dashboard as well as the functionality embedded in the dashboard. Political will and leadership are vital to ensure that deficiencies identified by the dashboard lead to better quality and more efficient and timely laboratory services.\nAs African laboratories move toward increasing the number of centers that prepare for or achieve accreditation[12], it is vital that laboratories are aware of the commitment needed to continually monitor, evaluate, and re-assess their status quo. Such commitment will ensure that the quality of the laboratory services they offer shows improvement over time. It is therefore important to consider what is required to achieve and maintain the quality of testing to avoid costly pitfalls[13] and inaccurate or delayed result reporting. In this regard, although much of the focus of quality management is placed on quality of tests themselves, time management in a laboratory is as crucial as assuring the quality of the tests performed. Without timely delivery of patient results, appropriate and meaningful clinical management of patients cannot be accomplished.\n\nLimitations \nThe data presented in this study focus on the within-laboratory network TAT and did not record or assess delays outside the laboratory capture net. Pre-analytical TAT referred to in this work denotes the time taken to transport a sample from a receiving laboratory to a testing laboratory. Ideally, sample tracking systems that relay tracking data to the central data warehouse, linked to discrete samples, will enable total end-to-end service assessment of TAT.\n\nLessons from the field \nThe dashboard subsequently developed has been extended to the top 22 highest volume tests performed across the organization but does not report data for pathology sections like microbiology or anatomical pathology disciplines, or the more specialized units like cytogenetics or immunology. Plans are underway to broaden the test basket and to additionally include critical tests such as cardiac troponin levels, shown in other work (not reported here) to have TAT that currently falls beyond meaningful clinical impact.\nThe data presented provide only a weekly snapshot. As technology permits, it is important to extend and broaden development of this dashboard at the database warehouse level using business intelligence analytics tools that enable reporting real-time data. It is envisaged that laboratories could use large screens within laboratories themselves to track real-time progress for immediate response and corrective action, where required. Alternatively, remote management could be facilitated using specially developed mobile devices to display live TAT performance.\nThe currently reported dashboard data does not distinguish between different levels of service (i.e., tertiary versus primary and secondary hospitals) with different levels of patient care (intensive care unit, STAT-lab, trauma departments). Data is aggregated and compared to the national cutoff for each test in the dashboard presented here. However, individual laboratories have established locally-relevant TAT cutoffs for emergency and routine contexts depending on the level of care (primary versus tertiary).\n\nAcknowledgements \nThe authors thank area, business, and laboratory managers in Gauteng for their participation in the pilot project. The test basket used in the development of this dashboard was determined in consultation with this group. The authors also thank Mr. Bahule Motlonye of the National Health Laboratory Service for his input during the pilot phase of the dashboard development undertaken during 2017.\n\nAuthor contributions \nD.K.G. supervised the study by providing leadership and oversight as the project leader. N.C., M.E.T. and L.M.C. designed the study, developed the methodology and conducted the research. M.E.T. developed the data systems to deliver a dashboard. M.E.T., N.C. and L.M.C. conducted the data analysis. D.K.G. reviewed the data, provided editorial comments and technical input. All authors contributed to the manuscript development.\n\nDisclaimer \nThe views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.\n\nCompeting interests \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 1.0 1.1 1.2 1.3 Hawkins, R.C. (2007). \"Laboratory turnaround time\". The Clinical Biochemist 28 (4): 179\u201394. PMC PMC2282400. PMID 18392122. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2282400 .   \n\n\u2191 \"TrakCare Lab Enterprise\". InterSystems. https:\/\/www.intersystems.com\/products\/trakcare\/trakcare-lab-enterprise . Retrieved 03 December 2018 .   \n\n\u2191 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Coetzee, L.-M.; Cassim, N.; Glencross, D.K. (2018). \"Using laboratory data to categorise CD4 laboratory turn-around-time performance across a national programme\". African Journal of Laboratory Medicine 7 (1): 665. doi:10.4102\/ajlm.v7i1.665. PMC PMC6111574. PMID 30167387. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6111574 .   \n\n\u2191 Drury, S.; Coetzee, L.-M.; Cassim, N. et al. (2012). \"Using Central Data Warehouse (CDW) Reports for Monitoring CD4 Laboratory Workload and Related Turn- Around-Time (TAT)\". Proceedings of the First International Conference of the African Society for Laboratory Medicine. doi:10.13140\/2.1.1277.7922.   \n\n\u2191 Chapple, M. (2018). \"Facts vs Dimensions Tables in a Database\". Lifewire. https:\/\/www.lifewire.com\/facts-vs-dimensions-1019646 . Retrieved 03 December 2018 .   \n\n\u2191 Carmona, S.; Macleod, W. (November 2016). \"Development of paediatric, VL and CD4 dashboards and results for action reports\" (PDF). Right To Care. http:\/\/www.righttocare.org\/wp-content\/uploads\/2016\/11\/Programme.pdf . Retrieved 03 December 2018 .   \n\n\u2191 \"Apps and services\". Microsoft. https:\/\/www.microsoft.com\/en-za\/microsoft-365\/products-apps-services . Retrieved 03 December 2018 .   \n\n\u2191 \"Download MicroStrategy Desktop\". MicroStrategy. https:\/\/www.microstrategy.com\/en\/get-started\/desktop . Retrieved 03 December 2018 .   \n\n\u2191 9.0 9.1 9.2 9.3 Cassim, N.; Coetzee, L.-M.; Tepper, M.E. et al. (2020). \"Timely delivery of laboratory efficiency information, Part II: Assessing the impact of a turn-around time dashboard at a high-volume laboratory\". African Journal of Laboratory Medicine 9 (2): a948. doi:10.4102\/ajlm.v9i2.948. PMC PMC7203269. PMID 32391245. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7203269 .   \n\n\u2191 Krasowski, M.D.; Davis, S.R.; Drees, D. et al. (2014). \"Autoverification in a core clinical chemistry laboratory at an academic medical center\". Journal of Pathology Informatics 5 (1): 13. doi:10.4103\/2153-3539.129450. PMC PMC4023033. PMID 24843824. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4023033 .   \n\n\u2191 11.0 11.1 Noble, M.A.; Martin, R.; Ndihokubwayo, J.-B. (2014). \"Making great strides in medical laboratory quality\". African Journal of Laboratory Medicine 3 (2): 256. doi:10.4102\/ajlm.v3i2.256. PMC PMC5637788. PMID 29043199. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5637788 .   \n\n\u2191 12.0 12.1 12.2 12.3 Nkengasong, J.N.; Birx, D. (2014). \"Quality matters in strengthening global laboratory medicine\". African Journal of Laboratory Medicine 3 (2): 239. doi:10.4102\/ajlm.v3i2.239. PMC PMC4956090. PMID 27453824. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4956090 .   \n\n\u2191 Okeke, I.N. (2018). \"Building resources to meet evolving laboratory medicine challenges in Africa\". African Journal of Laboratory Medicine 7 (1): 915. doi:10.4102\/ajlm.v7i1.915. PMC PMC6296021. PMID 30568895. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6296021 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The URL to Carmona and Mcleod's Development of paediatric, VL and CD4 dashboards and results for action reports is dead, and an archived or replacement version could not be found on the internet. The orginal reference #11 looks like a repeat of #9 and was not included for this version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\">https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on laboratory informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 24 January 2021, at 16:51.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 411 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","d11707b0ec9b9843d1f6332c3f2b2930_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Timely_delivery_of_laboratory_efficiency_information_Part_I_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Timely delivery of laboratory efficiency information, Part I: Developing an interactive turnaround time dashboard at a high-volume laboratory<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: Mean <a href=\"https:\/\/en.wikipedia.org\/wiki\/Turnaround_time\" class=\"extiw wiki-link\" title=\"wikipedia:Turnaround time\" data-key=\"ddf25c1267aa8e0b4ebe6301570d3bd4\">turnaround time<\/a> (TAT) <a href=\"https:\/\/www.limswiki.org\/index.php\/Reporting\" title=\"Reporting\" class=\"mw-disambig wiki-link\" data-key=\"c83685a5b9154b511ee113bbffedb2e5\">reporting<\/a> for testing <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratories<\/a> in a national network is typically static and not immediately available for meaningful corrective action and does not allow for test-by-test or site-by-site interrogation of individual laboratory performance.\n<\/p><p><b>Objective<\/b>: The aim of this study was to develop an easy-to-use, visual dashboard to report interactive graphical TAT data to provide a weekly snapshot of TAT efficiency.\n<\/p><p><b>Methods<\/b>: An interactive dashboard was developed by staff from the National Priority Programme and Central Data Warehouse of the National Health Laboratory Service in Johannesburg, South Africa, during 2018. Steps required to develop the dashboard were summarized in a flowchart. To illustrate the dashboard, one week of data from a busy laboratory for a specific set of tests was analyzed using annual performance plan TAT cutoffs. Data were extracted and prepared to deliver an aggregate extract, with statistical measures provided, including test volumes, global percentage of tests that were within TAT cutoffs, and percentile statistics.\n<\/p><p><b>Results<\/b>: Nine steps were used to develop the dashboard iteratively, with continuous feedback for each step. The <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_warehouse\" title=\"Data warehouse\" class=\"wiki-link\" data-key=\"ca506499cdf544371c0a0d549ff0e9ee\">data warehouse<\/a> environment conformed and stored <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS) data in two formats: (1) fact and (2) dimension. Queries were developed to generate an aggregate TAT data extract to create the dashboard. The dashboard successfully delivered weekly TAT reports.\n<\/p><p><b>Conclusion<\/b>: Implementation of a TAT dashboard can successfully enable the delivery of near real-time information and provide a weekly snapshot of efficiency in the form of TAT performance to identify and quantitate bottlenecks in service delivery.\n<\/p><p><b>Keywords<\/b>: turnaround time, laboratory efficiency, interactive dashboard, indicators, performance assessment\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Turnaround_time\" class=\"extiw wiki-link\" title=\"wikipedia:Turnaround time\" data-key=\"ddf25c1267aa8e0b4ebe6301570d3bd4\">Turnaround time<\/a> (TAT) is an important performance indicator for a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> service. It refers to the time from first registration in a laboratory to a result released in the <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS).<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-0\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup> Historically, within the National Health Laboratory Service (NHLS) of South Africa, TAT <a href=\"https:\/\/www.limswiki.org\/index.php\/Reporting\" title=\"Reporting\" class=\"mw-disambig wiki-link\" data-key=\"c83685a5b9154b511ee113bbffedb2e5\">reporting<\/a> was provided in annual and quarterly static management reports generated by the LIS, <a href=\"https:\/\/www.limswiki.org\/index.php\/InterSystems_Corporation\" title=\"InterSystems Corporation\" class=\"wiki-link\" data-key=\"148a499ad7e73769de328a0336ca2bab\">TrakCare<\/a><sup id=\"rdp-ebb-cite_ref-ISTrakCare_2-0\" class=\"reference\"><a href=\"#cite_note-ISTrakCare-2\">[2]<\/a><\/sup>, which also provided <i>ad hoc<\/i> reporting for use at the laboratory level. These reports are printed to provide a snapshot of TAT reporting and are suited for staff working at the laboratory level. At a national level, the corporate data warehouse (CDW) of the NHLS collated global TAT data from over 266 testing laboratories based on predetermined, annual performance plan cutoffs. National TAT cutoffs are set by expert committees of different pathology disciplines, with final confirmation from senior management before implementation. These cutoffs are set with provisions for all levels of the service laboratory: from low-volume laboratories with limited test repertoires to high-volume testing laboratories with extensive test offerings, including specialized testing, such as viral load testing. However, a large percentage of NHLS laboratories have 24-hour service and have emergency units in the <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospitals<\/a> in which they are housed; such laboratories have locally stricter TAT cutoffs for emergency and other local tests than are reflected in the national cutoffs for all <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">samples<\/a>.\n<\/p><p>Historically, the NHLS CDW TAT reports generated were static and reported only the mean TAT. Turnaround time data have a positive skewness, that is, a long tail to the right, meaning that the mean will be greater than the median. This implies that TAT data reported previously<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-1\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-0\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup>, reporting the mean TAT, masks good performance while concealing poor efficiency. Further, neither the current LIS nor CDW reports enable detailed analysis of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> or drilling down to laboratory- or test-level data for additional information about TAT efficiencies. Data presented at the first conference of the African Society for Laboratory Medicine in Cape Town, South Africa, in 2012, reported daily laboratory test volumes and mean TAT for authorized results, stratified by individual laboratories<sup id=\"rdp-ebb-cite_ref-DruryUsing12_4-0\" class=\"reference\"><a href=\"#cite_note-DruryUsing12-4\">[4]<\/a><\/sup>, providing a snapshot of performance. This enabled review of CD4 laboratory efficiency for a national program and provided important insights into laboratory operations.\n<\/p><p>A recent evaluation of TAT for HIV programs reported a methodology to further categorize laboratory TAT performance using three additional measures<sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-1\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup>: (1) median TAT, (2) 75th percentile TAT (tail size), and (3) percentage within cutoff TAT. These data were graphically presented using a scatter plot of percentage of samples within the TAT cutoff (<i>x<\/i>-axis) against the TAT 75th percentile (<i>y<\/i>-axis), categorized into four quadrants of performance to help identify the level of laboratory performance in a national program. This approach made it easier to identify both good performers and outliers in the same analysis.<sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-2\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup> The report was generated in Excel and included the raw monthly data, a scatter and bar graph, and a summary table of all laboratories per business region, the 75th percentile, the percentage within cutoff values, and TAT component 75th percentile values. This report was primarily distributed to managers of all HIV-related operational programs, that is, CD4, viral load and tuberculosis testing, and early infant diagnosis for review and intervention, and not shared across the network of testing laboratories.\n<\/p><p>Senior management in Gauteng, South Africa expressed a need for a TAT monitoring tool that would enable them to better manage their laboratories and identify sites with poor TAT performance. Given the static nature of the historic TAT reporting in the organization, an interactive system that offered information to enable review of performance, including outliers (tail size assessment), while confirming that sites were meeting cutoffs, would be a useful tool to enable business and laboratory managers alike to monitor their efficiency via TAT performance in real time. The concepts already developed and in use as Excel reports for the HIV programs were the starting point for developing a reporting dashboard for use across multiple disciplines and tests done throughout the network of 241 testing laboratories of the NHLS in South Africa.\n<\/p><p>The aim of this study was to develop an easy-to-use information system, in a dashboard format. This would enable weekly reporting of TAT data as a snapshot of performance. To achieve this, a number of changes to current TAT reporting had to be addressed. These changes included (1) moving from program-specific, single-test TAT reporting previously used<sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-3\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup> to a specific set of high-volume tests, (2) adopting TAT measures reported by Coetzee <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-4\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup>, (3) identifying dashboard software to use, and (4) identifying the target users.<sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-5\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup> The specific set of tests (or \"basket\" of commonly requested tests) should be representative across the primary pathology disciplines.\n<\/p><p>This article sets out to describe the process followed to develop the TAT dashboard\u2014using available software\u2014that could provide a weekly summary of national, business unit, and laboratory-level TAT performance for a basket of tests. For the purposes of this article, data from a single participating tertiary laboratory were used to illustrate the data distribution, as it represents an example of a testing facility that performed all tests reviewed in the prescribed national basket. This was done to show the respective levels of drilling functionality of the dashboard and to iterate the interactive properties, while demonstrating how the dashboard can be used to assess performance and identify outliers for intervention.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Ethical_considerations\">Ethical considerations<\/span><\/h3>\n<p>Ethics clearance for this study was obtained from the University of the Witwatersrand (M1706108). Only anonymized laboratory data, containing no patient identifiers, were used for the study.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Study_design\">Study design<\/span><\/h3>\n<p>A retrospective descriptive study design was used to analyze and report laboratory TAT data for a specific set of tests (Table 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Cassim_AfricanJLabMed2020_9-2.jpg\" class=\"image wiki-link\" data-key=\"9c69383d85745dd227df93f766247ea7\"><img alt=\"Tab1 Cassim AfricanJLabMed2020 9-2.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/40\/Tab1_Cassim_AfricanJLabMed2020_9-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 1.<\/b> Sample of a turnaround time dashboard table that lists the outcomes for the basket of tests, South Africa, 2018<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Steps_to_developing_a_turnaround_time_dashboard\">Steps to developing a turnaround time dashboard<\/span><\/h3>\n<p>The various steps required to develop the dashboard are summarized in a flowchart (Figure 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Cassim_AfricanJLabMed2020_9-2.jpg\" class=\"image wiki-link\" data-key=\"d7d12f135d8ad281470919880e0b7887\"><img alt=\"Fig1 Cassim AfricanJLabMed2020 9-2.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0c\/Fig1_Cassim_AfricanJLabMed2020_9-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1.<\/b> Flowchart depicting all steps required to develop a turnaround time dashboard, South Africa, 2018<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Sample_population_and_turnaround_time_definition\">Sample population and turnaround time definition<\/span><\/h3>\n<p>Using convenience sampling, data were selected from among the tests performed at a single busy academic laboratory in Gauteng for one week during 2018. Aside from global TAT reporting, the dashboard should adopt the three TAT measures reported by Coetzee '<i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-6\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup> These include pre-analytical, analytical, and post-analytical components of TAT components, namely: (1) the time from first registration at the source laboratory to registration of the referral at the testing laboratory (LAB-to-LAB TAT), (2) time from registration at the testing laboratory to results being populated by the LIS interface (TESTING TAT), and (3) time from result population by the LIS interface to manual review and authorization by senior laboratory staff (REVIEW TAT).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Test_basket_development_and_inclusion_and_exclusion_criteria\">Test basket development and inclusion and exclusion criteria<\/span><\/h3>\n<p>Focus group meetings were arranged with local area and business managers to define a test basket for the dashboard. The principles adopted were as follows: (1) measure a limited number of tests with a focus on the tests with the highest volumes of tests performed; (2) measure data for the indicator analyte (as a proxy) for specific panel tests (for example, the creatinine test was used as an indicator for assessment of urea and electrolyte test performance); (3) use the annual performance plan TAT cutoffs; and (4) deliver dashboard files via email (due to bandwidth constraints). All samples within this organization test basket were included in the example analysis and included the most commonly requested tests selected from <a href=\"https:\/\/www.limswiki.org\/index.php\/Hematology\" title=\"Hematology\" class=\"wiki-link\" data-key=\"de8b49c7b0be3cec33af362e763b9b0c\">hematology<\/a>, coagulation, HIV-tuberculosis, and chemistry (Table 1). A mapping table was developed to identify the LIS test sets and items to be reported. For each test, the TAT cutoff was also stipulated. The mapping table was used to guide the data extract.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_extraction\">Data extraction<\/span><\/h3>\n<p>For the purposes of demonstrating how the data were manipulated to create the dashboard, data were extracted for the week of September 2\u20138, 2018 from the CDW from four data sources: (1) the Operational Data Store that contained the original LIS data (Figure 2), (2) the \"CDW fact\" that reported test volumes, (3) the test method dimension<sup id=\"rdp-ebb-cite_ref-ChappleFacts18_5-0\" class=\"reference\"><a href=\"#cite_note-ChappleFacts18-5\">[5]<\/a><\/sup> (provides details on the test such as a unique identifier, discipline, test method code and name and national number from the CDW), and (4) TAT cutoff dimension (captures annual performance plan cutoffs) (Figure 1).<sup id=\"rdp-ebb-cite_ref-CarmonaDevelop16_6-0\" class=\"reference\"><a href=\"#cite_note-CarmonaDevelop16-6\">[6]<\/a><\/sup> Using an outer join, data from these four data sources were prepared as a temporary detailed table. The first temporary table limited data to the test basket, adding the TAT cutoffs and provided information using the laboratory hierarchy (region, business unit, and laboratory). Because this table would be too large to use for the dashboard, and assuming email delivery of the final report, two additional steps were used to create a smaller aggregate dataset. The mean, standard deviation, 75th percentile, and percentage within TAT cutoff were added. All TAT data were reported in hours. The final temporary table was exported as a Microsoft Excel (Redmond, Washington, United States)<sup id=\"rdp-ebb-cite_ref-Microsoft_7-0\" class=\"reference\"><a href=\"#cite_note-Microsoft-7\">[7]<\/a><\/sup> worksheet and imported into the MicroStrategy Desktop analytics tool (Providence, Virginia, United States).<sup id=\"rdp-ebb-cite_ref-MicroStratDesk_8-0\" class=\"reference\"><a href=\"#cite_note-MicroStratDesk-8\">[8]<\/a><\/sup> After the data were imported, the respective dashboard sheets were developed to include relevant TAT information for all levels of management.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Cassim_AfricanJLabMed2020_9-2.jpg\" class=\"image wiki-link\" data-key=\"02d25f7bb5beef64a3a172f187033afc\"><img alt=\"Fig2 Cassim AfricanJLabMed2020 9-2.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b7\/Fig2_Cassim_AfricanJLabMed2020_9-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2.<\/b> Visual representation of data preparation steps to transform and move raw turnaround time data, South Africa, 2018. Data were moved from the Operational Data Store (the laboratory information system) to a production server which consists of facts and dimensions. The production server structures the data by setting sets of data to specific target areas to facilitate final reporting in dashboard format.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Criteria_for_an_effective_dashboard\">Criteria for an effective dashboard<\/span><\/h3>\n<p>For any dashboard to be effective, it needs to adhere to a number of key outputs: it should (1) be visually engaging and easy to view and understand the TAT data displayed; (2) enable dynamic drilling down from a bird\u2019s-eye view to a local perspective, that is, from national or provincial level down to the laboratory level per test; (3) provide a report on a weekly basis for a TAT snapshot view; and (4) highlight TAT outliers for laboratory managers to follow-up and direct corrective action. From a more technical perspective, the dashboard also had to include additional features that included (1) conditional formatting to highlight good, average, and poor performance; (2) provision of various reporting formats such as bubble charts, tables, and bar charts; (3) the ability to import data for a variety of formats; and (4) theability to send the weekly dashboard data file via email in a small file format (\u2264 6 MB).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_analysis_and_visual_dashboard_display\">Data analysis and visual dashboard display<\/span><\/h3>\n<p>The dashboard displays (sheets) developed were as follows: (1) a bubble chart reporting the percentage within TAT cutoffs and 75th percentiles, (2) a table (see Table 1) displaying the bubble chart data, and (3) the 75th percentile for each phase of the component TAT reported by the test method. A bubble chart dashboard sheet was created to include: (1) the 75th percentile TAT (y-axis), (2) the percentage within TAT cutoff (x-axis), (3) the test volumes (size by and color by), and (4) the test method (color by and break by). The region codes and laboratory names were added to the dashboard as filters (radio buttons and search box display styles). An 85% within TAT cutoff reference line was added to aid identification of specific tests and associated laboratories with TAT that were outside of the TAT cutoffs. The data used to generate the bubble chart were also reported as a table in a separate sheet. The table listed the test name, total number of tests, TAT cutoff, the percentage within TAT cutoff, and 75th percentile TAT for the basket of tests reported on.\n<\/p><p>The table uses \"stop highlighting\" to denote the different percentage within TAT cutoff as follows: (1) 85% or higher in green, (2) 75% \u2013 84% in orange, and (3) under 75% in red. Lower percentage within TAT cutoff and higher 75th percentiles indicate an increased risk that any given laboratory is not adequately delivering patient reports that will enable timely clinical intervention.\n<\/p><p>A component TAT sheet was created as a clustered horizontal bar chart to display the component TAT with (1) test method name (<i>y<\/i>-axis) and (2) component 75th percentile TAT (LAB-LAB TAT, TESTING TAT and REVIEW TAT), differentiated by color. The testing laboratory name was added to enable refining and filtering data down to the laboratory level.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>The successfully developed dashboard enabled delivery of weekly TAT data. Data from 45,599 reported samples for the week of September 2\u20138, 2018 were utilized to demonstrate the dashboard development described here. The 75th percentile and the percentage of tests within stipulated cutoff for each test in the basket were visualized on the dashboard landing screen. This allowed the user to view data by test at the national, provincial and laboratory levels to visually identify outlying tests. The dashboard contained three individual sheets: the bubble chart, the TAT table, and the component TAT sheets.\n<\/p><p>Figure 3 shows typical weekly TAT data presentation outcomes as a bubble chart dashboard. In this example data set, only one test method, rapid plasma reagin (syphilis), failed to meet the 85% TAT cutoff and is reported as a small grey dot on the bubble chart dashboard. For this test, the reported percentage within cutoff was 83.8%, within the 75% and 84% category highlighted as orange in the prior Table 1. A cluster of tests in the bottom right reported 90% or higher within TAT cutoff with a 75th percentile TAT of eight hours or less. Only one test reported a percentage within TAT cutoff between 85% and 89%: total cholesterol (red dot). Higher test volumes were reported for the HIV viral load (<i>n<\/i> = 19,055) and creatinine (<i>n<\/i> = 8,857) tests.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Cassim_AfricanJLabMed2020_9-2.jpg\" class=\"image wiki-link\" data-key=\"ebee29e2aafebfb90cbe2779462c6f65\"><img alt=\"Fig3 Cassim AfricanJLabMed2020 9-2.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/1b\/Fig3_Cassim_AfricanJLabMed2020_9-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3.<\/b> An example of the Microstrategy Desktop bubble dashboard chart used to report total turnaround time data for an example site\u2019s week\u2019s data, South Africa, 2018. The percentage within cutoff turnaround time is reported on the <i>x<\/i>-axis with the 75th percentile turnaround time on the <i>y<\/i>-axis. The bubble size indicates test volumes. Reference lines were added at 85% within stipulated turnaround time cutoff on the <i>x<\/i>-axis. Each test within the test basket is colour coded with the key provided on the right. Outlying tests are immediately visible.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For the TAT table, results for the bubble chart are summarized (see Table 1) per test. At the 75th percentile TAT, no test exceeded the cutoff TAT. A 100% within cutoff TAT was reported for three tests: activated partial thromboplastin time, full blood count, and platelet count. Similarly, six tests reported a percentage within cutoff TAT between 95% and 99%.\n<\/p><p>The dashboard also reports component TAT in hours (Figure 4), including (1) LAB-TO-LAB, (2) TESTING, and (3) REVIEW times, with the tail size in hours for the distribution of each component TAT. In any given laboratory, some samples tested are local (from the immediately adjacent hospital), while other samples are referred for testing from nearby hospitals where these tests are not available. As such, a zero LAB-TO-LAB component indicates that the samples were not referred but are samples collected and tested locally. For referred samples included in the example data set (see Figure 4, CD4 antiretrovirals, D-Dimer, and HIV viral load, among others), the LAB-TO-LAB component TAT 75th percentile represents the inter-laboratory referral time, ranging in this instance from 12 to 23 hours (Figure 4). In the testing phase, TAT ranged from 0.25 to 63 hours (where 63 hours represented a single test, the rapid plasma reagin, syphilis, that was regarded locally as an outlier; see Table 1 for detail). The 75th percentile review TAT was two hours or less across all tests.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Cassim_AfricanJLabMed2020_9-2.jpg\" class=\"image wiki-link\" data-key=\"6e721b0701e214f14f72db363aaa6eab\"><img alt=\"Fig4 Cassim AfricanJLabMed2020 9-2.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/95\/Fig4_Cassim_AfricanJLabMed2020_9-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 4.<\/b> MicroStrategy Desktop dashboard bar chart used to report the component turnaround time data for an example site\u2019s for the week of September 2\u20138, 2018, South Africa. The components reported are LAB-LAB (inter-laboratory referral time), TESTING (time from registration to testing), and REVIEW (time from testing to review) turnaround time components.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>Access to information in an interactive dashboard format has previously enabled retrieval of health data for immediate clinical use in the NHLS in South Africa.<sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-7\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup> A similar approach has been applied and demonstrated in this work for TAT data. The dashboard described here provides an interactive, weekly snapshot of TAT performance, together with information about TAT distribution, tail size (outlier) assessment<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-2\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-8\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup>, to varying levels of laboratory managers across the NHLS, to enable timely intervention where poor service delivery is identified.\n<\/p><p>The dashboard is comprised of a few basic parameters that act together to provide information about TAT. Date stamping of samples in the LIS is a prerequisite to provide the basic information necessary to detail TAT linked to any given sample. Together with relevant sample identification datalogged, data is transferred to a central database for careful curation. Later, TAT data extraction is performed using standard data query tools. In the instance of a wider network of laboratories operating within the same organization, such as the South African NHLS, LIS data is stored using a decentralized architecture. Aggregate data, in the format described above, can then be collated and used to develop national TAT dashboards.\n<\/p><p>The dashboard described in this study simplifies presentation of complex data by enabling visualization of any given laboratory\u2019s efficiency. For the purpose of this study and to demonstrate the effectiveness and simple format of the dashboard developed, data from a single busy laboratory were used to illustrate the different outputs of the dashboard (graphs and table). The example data used here reveal how the dashboard can be used to identify tests that are not meeting national (or local) cutoff criteria. In the example presented, rapid plasma reagin (syphilis) testing was noted as an outlier, as it did not meet the organization-stipulated 85% within cutoff TAT. The summary table (example shown in Table 1) also provides a spreadsheet format table of the relevant tests either meeting, or failing to meet, the national cutoff criteria. The additional information on TAT component analysis further assists management to identify those areas of laboratory testing, within the respective pre-analytical, analytical, and post-analytical components, that may need investigation for improvement.\n<\/p><p>The dashboard was successfully rolled out to all NHLS testing laboratories; weekly data are currently received by these laboratories for review. The dashboard development included a drill-down function into the performance of a particular test to see results by testing laboratory, or business unit. The addition of tail size measures<sup id=\"rdp-ebb-cite_ref-HawkinsLab07_1-3\" class=\"reference\"><a href=\"#cite_note-HawkinsLab07-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CoetzeeUsing18_3-9\" class=\"reference\"><a href=\"#cite_note-CoetzeeUsing18-3\">[3]<\/a><\/sup> has also enabled managers to identify less efficient areas of their laboratory services with outlying performance, seen in the example case described in this article (rapid plasma reagin \u2013 syphilis), which would have been otherwise missed using conventional reporting alone (using mean TAT reporting). In addition, to enable practical sample-by-sample audit, individual samples that did not meet cutoff criteria were identified for follow-up in an additional summary table sheet (added at the request of laboratory managers to enable better intervention; not shown). Use of the dashboard has also led to laboratory process changes with improved individual component TAT. For example, post-analytical TAT improvements included implementation of an \"auto review\" feature.<sup id=\"rdp-ebb-cite_ref-CassimTimely20_9-0\" class=\"reference\"><a href=\"#cite_note-CassimTimely20-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KrasowskiAuto14_10-0\" class=\"reference\"><a href=\"#cite_note-KrasowskiAuto14-10\">[10]<\/a><\/sup> With respect to analytical delays identified, testing delays could be correlated with instrument breakdown logs from the laboratories or instrument suppliers to identify reasons for prolonged testing TAT.<sup id=\"rdp-ebb-cite_ref-CassimTimely20_9-1\" class=\"reference\"><a href=\"#cite_note-CassimTimely20-9\">[9]<\/a><\/sup> The impact of dashboard usage on improving TAT is described in detail in the companion article in this issue.<sup id=\"rdp-ebb-cite_ref-CassimTimely20_9-2\" class=\"reference\"><a href=\"#cite_note-CassimTimely20-9\">[9]<\/a><\/sup>\n<\/p><p>Risk management teaches that not all errors can be predicted.<sup id=\"rdp-ebb-cite_ref-NobleMaking14_11-0\" class=\"reference\"><a href=\"#cite_note-NobleMaking14-11\">[11]<\/a><\/sup> However, it is only through active review of quality processes that delays, errors, and problems can be detected earlier to enable corrective action. Thus, critical to managing risk is the continuous and ongoing evaluation and assessment of procedures and processes to ensure that the same errors are not repeated. Here, human capital is key to the sustainability and success of any dashboard implementation. Noble <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-NobleMaking14_11-1\" class=\"reference\"><a href=\"#cite_note-NobleMaking14-11\">[11]<\/a><\/sup> reported that only the persistence and interest of laboratory personnel to maintain quality can ensure smooth and rapid progress of error detection (and correction back to quality) that is fast and sustainable. \n<\/p><p>One of the fundamental lessons learned from the development of the dashboard described here is that providing tools to assess TAT performance does not in itself imply corrective action or improvement. The dashboard is merely a tool that enables managers to effectively and efficiently ensure procedural excellence. Nkengasong and Birx<sup id=\"rdp-ebb-cite_ref-NkengasongQual14_12-0\" class=\"reference\"><a href=\"#cite_note-NkengasongQual14-12\">[12]<\/a><\/sup> also suggest that in order for innovation to be adopted, and sustainable innovation and performance enablers should both energize and incentivize laboratories across four pillars: implementation, measurement, reward, and improvement. A culture of diligence and willingness on the part of managers to meaningfully use information provided in the dashboard is thus important to enable making consequential changes at the laboratory level. Political will and strong senior leadership are also needed to make systems, such as those introduced with the dashboard described here, both functional and sustainable.<sup id=\"rdp-ebb-cite_ref-NkengasongQual14_12-1\" class=\"reference\"><a href=\"#cite_note-NkengasongQual14-12\">[12]<\/a><\/sup> This can be done by appropriately recognizing and rewarding laboratories and personnel who use the tools provided.\n<\/p><p>Pre-analytical errors should not be underestimated, as they can increase both testing errors and TAT.<sup id=\"rdp-ebb-cite_ref-NkengasongQual14_12-2\" class=\"reference\"><a href=\"#cite_note-NkengasongQual14-12\">[12]<\/a><\/sup> In the example laboratory performance reported here, all four referred test TAT outcomes were compromised due to pre-analytical delays. Documenting these delays and acting to reduce pre-analytical time, including travel time, and time spent in receiving centers prior to sample registration, can be used to streamline services.\n<\/p><p>Another outcome reported by managers using the dashboard was that the information could be documented week-by-week to provide objective evidence to document and motivate for additional resources required to achieve TAT cutoffs, for example including additional sample collection schedules, increasing testing capacity, and providing motivation for auto-review and authorization modules.<sup id=\"rdp-ebb-cite_ref-CassimTimely20_9-3\" class=\"reference\"><a href=\"#cite_note-CassimTimely20-9\">[9]<\/a><\/sup>\n<\/p><p>It is important in the context of a resource-poor setting to highlight that the dashboard described here was developed without specific funding, relying only on the collaborative effort of NHLS staff (the authors) with data management or MicroStrategy skills. Data is routinely transferred from the LIS to the CDW, where is it collated and carefully curated for downstream research and operational needs. Initial formats were undertaken using CDW-extracted data analyzed in MS Excel to create simple charts plotting 75th percentile and median TAT, by laboratory, for annualized or quarterly aggregated TAT data. Thereafter, analyses were extended to create week-by-week practical and usable worksheets so that individual laboratories could view current data. Using MicroStrategy, a freely available software program, a dashboard was developed to enable automatic presentation of the data in a visible interactive format (with the snapshot aggregate data file emailed to users weekly) to facilitate automated more immediate access to current TAT data. Future planning includes providing live data in the dashboard, facilitated by extending local bandwidth capacity and immediate real-time analysis of data within the CDW itself.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>This article outlines the database management and methods used for the development of a dashboard that enables presentation of weekly TAT data to relevant business and laboratory managers, as part of the overall quality management portfolio of the organization. This novel approach ensures the delivery of quality, timely pathology reporting by the South African NHLS and, ultimately, better patient care. Training on the use of the dashboard is essential to ensure that users are competent. Users need to both understand the principles applied in the dashboard as well as the functionality embedded in the dashboard. Political will and leadership are vital to ensure that deficiencies identified by the dashboard lead to better quality and more efficient and timely laboratory services.\n<\/p><p>As African laboratories move toward increasing the number of centers that prepare for or achieve accreditation<sup id=\"rdp-ebb-cite_ref-NkengasongQual14_12-3\" class=\"reference\"><a href=\"#cite_note-NkengasongQual14-12\">[12]<\/a><\/sup>, it is vital that laboratories are aware of the commitment needed to continually monitor, evaluate, and re-assess their status quo. Such commitment will ensure that the quality of the laboratory services they offer shows improvement over time. It is therefore important to consider what is required to achieve and maintain the quality of testing to avoid costly pitfalls<sup id=\"rdp-ebb-cite_ref-OkekeBuild18_13-0\" class=\"reference\"><a href=\"#cite_note-OkekeBuild18-13\">[13]<\/a><\/sup> and inaccurate or delayed result reporting. In this regard, although much of the focus of quality management is placed on quality of tests themselves, time management in a laboratory is as crucial as assuring the quality of the tests performed. Without timely delivery of patient results, appropriate and meaningful clinical management of patients cannot be accomplished.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Limitations\">Limitations<\/span><\/h3>\n<p>The data presented in this study focus on the within-laboratory network TAT and did not record or assess delays outside the laboratory capture net. Pre-analytical TAT referred to in this work denotes the time taken to transport a sample from a receiving laboratory to a testing laboratory. Ideally, sample tracking systems that relay tracking data to the central data warehouse, linked to discrete samples, will enable total end-to-end service assessment of TAT.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Lessons_from_the_field\">Lessons from the field<\/span><\/h3>\n<p>The dashboard subsequently developed has been extended to the top 22 highest volume tests performed across the organization but does not report data for pathology sections like microbiology or anatomical pathology disciplines, or the more specialized units like cytogenetics or immunology. Plans are underway to broaden the test basket and to additionally include critical tests such as cardiac troponin levels, shown in other work (not reported here) to have TAT that currently falls beyond meaningful clinical impact.\n<\/p><p>The data presented provide only a weekly snapshot. As technology permits, it is important to extend and broaden development of this dashboard at the database warehouse level using business intelligence analytics tools that enable reporting real-time data. It is envisaged that laboratories could use large screens within laboratories themselves to track real-time progress for immediate response and corrective action, where required. Alternatively, remote management could be facilitated using specially developed mobile devices to display live TAT performance.\n<\/p><p>The currently reported dashboard data does not distinguish between different levels of service (i.e., tertiary versus primary and secondary hospitals) with different levels of patient care (intensive care unit, STAT-lab, trauma departments). Data is aggregated and compared to the national cutoff for each test in the dashboard presented here. However, individual laboratories have established locally-relevant TAT cutoffs for emergency and routine contexts depending on the level of care (primary versus tertiary).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors thank area, business, and laboratory managers in Gauteng for their participation in the pilot project. The test basket used in the development of this dashboard was determined in consultation with this group. The authors also thank Mr. Bahule Motlonye of the National Health Laboratory Service for his input during the pilot phase of the dashboard development undertaken during 2017.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>D.K.G. supervised the study by providing leadership and oversight as the project leader. N.C., M.E.T. and L.M.C. designed the study, developed the methodology and conducted the research. M.E.T. developed the data systems to deliver a dashboard. M.E.T., N.C. and L.M.C. conducted the data analysis. D.K.G. reviewed the data, provided editorial comments and technical input. All authors contributed to the manuscript development.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Disclaimer\">Disclaimer<\/span><\/h3>\n<p>The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HawkinsLab07-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HawkinsLab07_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-HawkinsLab07_1-1\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-HawkinsLab07_1-2\">1.2<\/a><\/sup> <sup><a href=\"#cite_ref-HawkinsLab07_1-3\">1.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hawkins, R.C. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2282400\" target=\"_blank\">\"Laboratory turnaround time\"<\/a>. <i>The Clinical Biochemist<\/i> <b>28<\/b> (4): 179\u201394. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2282400\/\" target=\"_blank\">PMC2282400<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18392122\" target=\"_blank\">18392122<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2282400\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2282400<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+turnaround+time&rft.jtitle=The+Clinical+Biochemist&rft.aulast=Hawkins%2C+R.C.&rft.au=Hawkins%2C+R.C.&rft.date=2007&rft.volume=28&rft.issue=4&rft.pages=179%E2%80%9394&rft_id=info:pmc\/PMC2282400&rft_id=info:pmid\/18392122&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2282400&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISTrakCare-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISTrakCare_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.intersystems.com\/products\/trakcare\/trakcare-lab-enterprise\" target=\"_blank\">\"TrakCare Lab Enterprise\"<\/a>. InterSystems<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.intersystems.com\/products\/trakcare\/trakcare-lab-enterprise\" target=\"_blank\">https:\/\/www.intersystems.com\/products\/trakcare\/trakcare-lab-enterprise<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=TrakCare+Lab+Enterprise&rft.atitle=&rft.pub=InterSystems&rft_id=https%3A%2F%2Fwww.intersystems.com%2Fproducts%2Ftrakcare%2Ftrakcare-lab-enterprise&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoetzeeUsing18-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CoetzeeUsing18_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-1\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-2\">3.2<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-3\">3.3<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-4\">3.4<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-5\">3.5<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-6\">3.6<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-7\">3.7<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-8\">3.8<\/a><\/sup> <sup><a href=\"#cite_ref-CoetzeeUsing18_3-9\">3.9<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coetzee, L.-M.; Cassim, N.; Glencross, D.K. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6111574\" target=\"_blank\">\"Using laboratory data to categorise CD4 laboratory turn-around-time performance across a national programme\"<\/a>. <i>African Journal of Laboratory Medicine<\/i> <b>7<\/b> (1): 665. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v7i1.665\" target=\"_blank\">10.4102\/ajlm.v7i1.665<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6111574\/\" target=\"_blank\">PMC6111574<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30167387\" target=\"_blank\">30167387<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6111574\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6111574<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+laboratory+data+to+categorise+CD4+laboratory+turn-around-time+performance+across+a+national+programme&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Coetzee%2C+L.-M.%3B+Cassim%2C+N.%3B+Glencross%2C+D.K.&rft.au=Coetzee%2C+L.-M.%3B+Cassim%2C+N.%3B+Glencross%2C+D.K.&rft.date=2018&rft.volume=7&rft.issue=1&rft.pages=665&rft_id=info:doi\/10.4102%2Fajlm.v7i1.665&rft_id=info:pmc\/PMC6111574&rft_id=info:pmid\/30167387&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6111574&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DruryUsing12-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DruryUsing12_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Drury, S.; Coetzee, L.-M.; Cassim, N. et al. (2012). \"Using Central Data Warehouse (CDW) Reports for Monitoring CD4 Laboratory Workload and Related Turn- Around-Time (TAT)\". <i>Proceedings of the First International Conference of the African Society for Laboratory Medicine<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13140%2F2.1.1277.7922\" target=\"_blank\">10.13140\/2.1.1277.7922<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+Central+Data+Warehouse+%28CDW%29+Reports+for+Monitoring+CD4+Laboratory+Workload+and+Related+Turn-+Around-Time+%28TAT%29&rft.jtitle=Proceedings+of+the+First+International+Conference+of+the+African+Society+for+Laboratory+Medicine&rft.aulast=Drury%2C+S.%3B+Coetzee%2C+L.-M.%3B+Cassim%2C+N.+et+al.&rft.au=Drury%2C+S.%3B+Coetzee%2C+L.-M.%3B+Cassim%2C+N.+et+al.&rft.date=2012&rft_id=info:doi\/10.13140%2F2.1.1277.7922&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChappleFacts18-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChappleFacts18_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Chapple, M. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.lifewire.com\/facts-vs-dimensions-1019646\" target=\"_blank\">\"Facts vs Dimensions Tables in a Database\"<\/a>. <i>Lifewire<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.lifewire.com\/facts-vs-dimensions-1019646\" target=\"_blank\">https:\/\/www.lifewire.com\/facts-vs-dimensions-1019646<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Facts+vs+Dimensions+Tables+in+a+Database&rft.atitle=Lifewire&rft.aulast=Chapple%2C+M.&rft.au=Chapple%2C+M.&rft.date=2018&rft_id=https%3A%2F%2Fwww.lifewire.com%2Ffacts-vs-dimensions-1019646&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarmonaDevelop16-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CarmonaDevelop16_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Carmona, S.; Macleod, W. (November 2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.righttocare.org\/wp-content\/uploads\/2016\/11\/Programme.pdf\" target=\"_blank\">\"Development of paediatric, VL and CD4 dashboards and results for action reports\"<\/a> (PDF). <i>Right To Care<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.righttocare.org\/wp-content\/uploads\/2016\/11\/Programme.pdf\" target=\"_blank\">http:\/\/www.righttocare.org\/wp-content\/uploads\/2016\/11\/Programme.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Development+of+paediatric%2C+VL+and+CD4+dashboards+and+results+for+action+reports&rft.atitle=Right+To+Care&rft.aulast=Carmona%2C+S.%3B+Macleod%2C+W.&rft.au=Carmona%2C+S.%3B+Macleod%2C+W.&rft.date=November+2016&rft_id=http%3A%2F%2Fwww.righttocare.org%2Fwp-content%2Fuploads%2F2016%2F11%2FProgramme.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Microsoft-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Microsoft_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.microsoft.com\/en-za\/microsoft-365\/products-apps-services\" target=\"_blank\">\"Apps and services\"<\/a>. Microsoft<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.microsoft.com\/en-za\/microsoft-365\/products-apps-services\" target=\"_blank\">https:\/\/www.microsoft.com\/en-za\/microsoft-365\/products-apps-services<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Apps+and+services&rft.atitle=&rft.pub=Microsoft&rft_id=https%3A%2F%2Fwww.microsoft.com%2Fen-za%2Fmicrosoft-365%2Fproducts-apps-services&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MicroStratDesk-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MicroStratDesk_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.microstrategy.com\/en\/get-started\/desktop\" target=\"_blank\">\"Download MicroStrategy Desktop\"<\/a>. MicroStrategy<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.microstrategy.com\/en\/get-started\/desktop\" target=\"_blank\">https:\/\/www.microstrategy.com\/en\/get-started\/desktop<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Download+MicroStrategy+Desktop&rft.atitle=&rft.pub=MicroStrategy&rft_id=https%3A%2F%2Fwww.microstrategy.com%2Fen%2Fget-started%2Fdesktop&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CassimTimely20-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CassimTimely20_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-CassimTimely20_9-1\">9.1<\/a><\/sup> <sup><a href=\"#cite_ref-CassimTimely20_9-2\">9.2<\/a><\/sup> <sup><a href=\"#cite_ref-CassimTimely20_9-3\">9.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cassim, N.; Coetzee, L.-M.; Tepper, M.E. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7203269\" target=\"_blank\">\"Timely delivery of laboratory efficiency information, Part II: Assessing the impact of a turn-around time dashboard at a high-volume laboratory\"<\/a>. <i>African Journal of Laboratory Medicine<\/i> <b>9<\/b> (2): a948. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v9i2.948\" target=\"_blank\">10.4102\/ajlm.v9i2.948<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7203269\/\" target=\"_blank\">PMC7203269<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32391245\" target=\"_blank\">32391245<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7203269\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7203269<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Timely+delivery+of+laboratory+efficiency+information%2C+Part+II%3A+Assessing+the+impact+of+a+turn-around+time+dashboard+at+a+high-volume+laboratory&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Cassim%2C+N.%3B+Coetzee%2C+L.-M.%3B+Tepper%2C+M.E.+et+al.&rft.au=Cassim%2C+N.%3B+Coetzee%2C+L.-M.%3B+Tepper%2C+M.E.+et+al.&rft.date=2020&rft.volume=9&rft.issue=2&rft.pages=a948&rft_id=info:doi\/10.4102%2Fajlm.v9i2.948&rft_id=info:pmc\/PMC7203269&rft_id=info:pmid\/32391245&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7203269&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrasowskiAuto14-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrasowskiAuto14_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krasowski, M.D.; Davis, S.R.; Drees, D. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4023033\" target=\"_blank\">\"Autoverification in a core clinical chemistry laboratory at an academic medical center\"<\/a>. <i>Journal of Pathology Informatics<\/i> <b>5<\/b> (1): 13. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2F2153-3539.129450\" target=\"_blank\">10.4103\/2153-3539.129450<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4023033\/\" target=\"_blank\">PMC4023033<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24843824\" target=\"_blank\">24843824<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4023033\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4023033<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Autoverification+in+a+core+clinical+chemistry+laboratory+at+an+academic+medical+center&rft.jtitle=Journal+of+Pathology+Informatics&rft.aulast=Krasowski%2C+M.D.%3B+Davis%2C+S.R.%3B+Drees%2C+D.+et+al.&rft.au=Krasowski%2C+M.D.%3B+Davis%2C+S.R.%3B+Drees%2C+D.+et+al.&rft.date=2014&rft.volume=5&rft.issue=1&rft.pages=13&rft_id=info:doi\/10.4103%2F2153-3539.129450&rft_id=info:pmc\/PMC4023033&rft_id=info:pmid\/24843824&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4023033&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NobleMaking14-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NobleMaking14_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-NobleMaking14_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Noble, M.A.; Martin, R.; Ndihokubwayo, J.-B. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5637788\" target=\"_blank\">\"Making great strides in medical laboratory quality\"<\/a>. <i>African Journal of Laboratory Medicine<\/i> <b>3<\/b> (2): 256. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v3i2.256\" target=\"_blank\">10.4102\/ajlm.v3i2.256<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5637788\/\" target=\"_blank\">PMC5637788<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29043199\" target=\"_blank\">29043199<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5637788\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5637788<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Making+great+strides+in+medical+laboratory+quality&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Noble%2C+M.A.%3B+Martin%2C+R.%3B+Ndihokubwayo%2C+J.-B.&rft.au=Noble%2C+M.A.%3B+Martin%2C+R.%3B+Ndihokubwayo%2C+J.-B.&rft.date=2014&rft.volume=3&rft.issue=2&rft.pages=256&rft_id=info:doi\/10.4102%2Fajlm.v3i2.256&rft_id=info:pmc\/PMC5637788&rft_id=info:pmid\/29043199&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5637788&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NkengasongQual14-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NkengasongQual14_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-NkengasongQual14_12-1\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-NkengasongQual14_12-2\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-NkengasongQual14_12-3\">12.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nkengasong, J.N.; Birx, D. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4956090\" target=\"_blank\">\"Quality matters in strengthening global laboratory medicine\"<\/a>. <i>African Journal of Laboratory Medicine<\/i> <b>3<\/b> (2): 239. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v3i2.239\" target=\"_blank\">10.4102\/ajlm.v3i2.239<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4956090\/\" target=\"_blank\">PMC4956090<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27453824\" target=\"_blank\">27453824<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4956090\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4956090<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quality+matters+in+strengthening+global+laboratory+medicine&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Nkengasong%2C+J.N.%3B+Birx%2C+D.&rft.au=Nkengasong%2C+J.N.%3B+Birx%2C+D.&rft.date=2014&rft.volume=3&rft.issue=2&rft.pages=239&rft_id=info:doi\/10.4102%2Fajlm.v3i2.239&rft_id=info:pmc\/PMC4956090&rft_id=info:pmid\/27453824&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4956090&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OkekeBuild18-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OkekeBuild18_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Okeke, I.N. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6296021\" target=\"_blank\">\"Building resources to meet evolving laboratory medicine challenges in Africa\"<\/a>. <i>African Journal of Laboratory Medicine<\/i> <b>7<\/b> (1): 915. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4102%2Fajlm.v7i1.915\" target=\"_blank\">10.4102\/ajlm.v7i1.915<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6296021\/\" target=\"_blank\">PMC6296021<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30568895\" target=\"_blank\">30568895<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6296021\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6296021<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+resources+to+meet+evolving+laboratory+medicine+challenges+in+Africa&rft.jtitle=African+Journal+of+Laboratory+Medicine&rft.aulast=Okeke%2C+I.N.&rft.au=Okeke%2C+I.N.&rft.date=2018&rft.volume=7&rft.issue=1&rft.pages=915&rft_id=info:doi\/10.4102%2Fajlm.v7i1.915&rft_id=info:pmc\/PMC6296021&rft_id=info:pmid\/30568895&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6296021&rfr_id=info:sid\/en.wikipedia.org:Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The URL to Carmona and Mcleod's <i> Development of paediatric, VL and CD4 dashboards and results for action reports<\/i> is dead, and an archived or replacement version could not be found on the internet. The orginal reference #11 looks like a repeat of #9 and was not included for this version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194113\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.449 seconds\nReal time usage: 1.989 seconds\nPreprocessor visited node count: 11789\/1000000\nPreprocessor generated node count: 31782\/1000000\nPost\u2010expand include size: 96777\/2097152 bytes\nTemplate argument size: 31960\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 260.311 1 - -total\n 75.33% 196.105 1 - Template:Reflist\n 61.69% 160.573 13 - Template:Citation\/core\n 46.13% 120.082 8 - Template:Cite_journal\n 19.08% 49.668 5 - Template:Cite_web\n 14.69% 38.240 1 - Template:Infobox_journal_article\n 13.91% 36.204 1 - Template:Infobox\n 9.43% 24.556 80 - Template:Infobox\/row\n 7.71% 20.064 21 - Template:Citation\/identifier\n 4.65% 12.113 15 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12296-0!*!0!!en!5!* and timestamp 20210429194111 and revision id 41298\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory\">https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","d11707b0ec9b9843d1f6332c3f2b2930_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/40\/Tab1_Cassim_AfricanJLabMed2020_9-2.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0c\/Fig1_Cassim_AfricanJLabMed2020_9-2.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b7\/Fig2_Cassim_AfricanJLabMed2020_9-2.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/1b\/Fig3_Cassim_AfricanJLabMed2020_9-2.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/95\/Fig4_Cassim_AfricanJLabMed2020_9-2.jpg"],"d11707b0ec9b9843d1f6332c3f2b2930_timestamp":1619725270,"b18b6bb1bbee57c49590e7b3ba53097d_type":"article","b18b6bb1bbee57c49590e7b3ba53097d_title":"Digital transformation risk management in forensic science laboratories (Casey and Souvignet 2020)","b18b6bb1bbee57c49590e7b3ba53097d_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Digital_transformation_risk_management_in_forensic_science_laboratories","b18b6bb1bbee57c49590e7b3ba53097d_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Digital transformation risk management in forensic science laboratories\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDigital transformation risk management in forensic science laboratoriesJournal\n \nForensic Science InternationalAuthor(s)\n \nCasey, Eoghan; Souvignet, Thomas R.Author affiliation(s)\n \nUniversity of LausannePrimary contact\n \nEmail: thomas dot souvignet at unil dot chYear published\n \n2020Volume and issue\n \n316Article #\n \n110486DOI\n \n10.1016\/j.forsciint.2020.110486ISSN\n \n0379-0738Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.sciencedirect.com\/science\/article\/pii\/S0379073820303480Download\n \nhttps:\/\/www.sciencedirect.com\/science\/article\/pii\/S0379073820303480\/pdfft (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Risks and remedies \n\n3.1 Data retention \n\n3.1.1 Data loss scenario \n\n\n3.2 Evidence integrity \n\n3.2.1 Data alteration scenario \n\n\n3.3 Data traceability \n\n3.3.1 LIMS weaknesses scenario \n\n\n3.4 Computer system malfunction \n\n3.4.1 Hardware issues scenario \n\n\n3.5 Automation complexity and pitfalls \n\n3.5.1 Reliability and human rights concerns scenario \n\n\n\n\n4 Digital reinforcement of forensic science principles \n\n4.1 Authenticity and integrity \n\n4.1.1 Contextual reconstruction scenario \n\n\n4.2 Reliability and reproducibility \n\n4.2.1 Reproducibility of forensic analysis scenario \n\n\n4.3 Quality and efficiency \n\n4.3.1 Validation and change control scenario \n\n\n4.4 Forensic-by-design automation \n\n4.4.1 Forensic artificial intelligence scenario \n\n\n\n\n5 Digital transformation risk management \n\n5.1 Curated digital information \n5.2 Established processes and procedures \n5.3 Data integrity \n5.4 Strategic audit logging \n5.5 Data authentication \n5.6 Regular use or review of data and processes \n5.7 Test and validation \n5.8 Effective use of automation \n\n\n6 Role of digital forensic expertise \n7 Quality assurance considerations \n\n7.1 Current requirements \n7.2 Possible improvements \n\n\n8 Conclusions \n9 Acknowledgements \n\n9.1 Author contributions \n9.2 Conflicts of interest \n\n\n10 References \n11 Notes \n\n\n\nAbstract \nTechnological advances are changing how forensic laboratories operate in all forensic disciplines, not only digital. Computers support workflow management and enable evidence analysis (physical and digital), while new technology enables previously unavailable forensic capabilities. Used properly, the integration of digital systems supports greater efficiency and reproducibility, and drives digital transformation of forensic laboratories. However, without the necessary preparations, these digital transformations can undermine the core principles and processes of forensic laboratories. Forensic preparedness concentrating on digital data reduces the cost and operational disruption of responding to various kinds of problems, including misplaced exhibits, allegations of employee misconduct, disclosure requirements, and information security breaches. \nThis work gives pertinent examples of problems and risks involving technology that have occurred in forensic laboratories, along with opportunities and risk mitigation strategies, based on the authors\u2019 experiences. It also presents recommendations to help forensic laboratories prepare for and manage these risks, to use technology effectively, and ultimately strengthen forensic science. The importance of involving digital forensic expertise in risk management of digital transformations in laboratories is emphasized. Forensic laboratories that do not adopt forensic digital preparedness will produce results based on digital data and processes that cannot be verified independently, leaving them vulnerable to challenge. The recommendations in this work could enhance international standards such as ISO\/IEC 17025, which are used to assess and accredit laboratories.\nKeywords: forensic science, digital transformations, forensic laboratories, forensic preparedness, forensic digital preparedness, risk management, ISO\/IEC 17025, laboratory information management systems\n\nIntroduction \nForensic science laboratories are becoming more reliant on computers and data for both administrative and analytical operations. These technological advances create new opportunities and risks for all forensic disciplines, not only to digital evidence.[1] With proper preparation and management, forensic laboratories can employ technology effectively to improve performance and quality, while mitigating the associated risks. However, many forensic laboratories do not understand the subtlety and expertise required to manage risks of digital transformation, inadvisedly treating it as simply a technical component of existing quality management processes. Forensic laboratories that fail to realize the need for forensic digital preparedness to actively manage risks associated with digital transformations are vulnerable to significant expense, disruption, and liability when problems arise.\nForensic laboratories rely on technology for much more than communication and routine business functions. Sophisticated equipment for processing chemical and biological materials are operated using computers and save results in digital form. Mass spectrometers, DNA analysis systems, and other laboratory equipment save their results in raw data files. Digital evidence is processed using specialized hardware and software, although not all forensic laboratories have integrated this new discipline. Forensic laboratories are using computerized case management systems for tracking treatment of all evidential exhibits and forensic results. Automated systems with artificial intelligence (AI) are being used to support forensic analysis. In reality, digital transformations\u2014the use of digital technology to make existing processes more efficient and effective, and to develop new solutions to emerging problems\u2014are well underway, and forensic laboratories require a robust strategy to manage the associated risks and realize the opportunities.\nThis increased dependence on digital technology creates risks and opportunities for forensic laboratories. Potential pitfalls include loss of data needed to perform forensic analysis, errors in analysis of physical traces (e.g., DNA, fingerprint, face) caused by computer hardware or software, ability to tamper with raw data files generated by laboratory equipment, and incorrect information input into laboratory information management systems (LIMS). Possible benefits are traceability and integrity of traces, reliability and reproducibility of results from information extracted from traces and stored as raw data, and use of AI to support forensic analysis.\nLessons can be learned from the digital forensic domain, including forensic digital preparedness and accreditation challenges. Primary challenges encountered by digital forensic laboratories adopting quality standards include[2]:\n\n Inaccurate or insufficient information in technical records, including chain of custody, and no mechanism to detect subsequent changes to records.\n Problems with the security of information technology systems and the backup processes of data.\n Missing or insufficiently detailed procedures for treating digital data, and personnel not following documented procedures consistently.\n Lack of robust quality checking mechanisms, and issues with validation of methods.\nThis paper presents risks and opportunities associated with digital transformation of forensic laboratories, providing examples based on the authors\u2019 experiences. Examples have been anonymized, as the intention is to illustrate general lessons learned rather than critique specific laboratories. This work then presents forensic digital preparedness, a set of recommendations to help laboratories navigate risks associated with digital transformations, including mishandled exhibits, allegations of employee misconduct, and disclosure requirements. The role of digital forensic capabilities and expertise in risk management of digital transformations in laboratories is discussed. This work culminates with broader implications for international standards such as ISO\/IEC 17025, which are used to assess and accredit laboratories.\n\nRisks and remedies \nMany processes in forensic laboratories have become digitalized through the increased use of information management systems and software running analysis instruments. While these systems serve crucial functions in modern forensic laboratories, thet also have associated risks that must be managed.\n\nData retention \nThe computer systems used to store instruments' generated data files (raw and processed) can encounter problems that lead to loss of information.\n\nData loss scenario \nIn this scenario, Reust et al.[3] presented a case study concerning a forensic laboratory that performed DNA analysis of a crime scene sample relevant to a multiple homicide and death penalty case, but did not retain a copy of the raw data files. To comply with a court order to provide the defense with original raw data, it was necessary to perform costly forensic data recovery on the computer used to perform the original processing of DNA. The authors developed a customized software utility to automatically search the computer hard drive for all fragments of the relevant raw data and reconstruct the original files. The resulting files were tested and validated with DNA analysis software.\nAs seen with Reust et al., original data files thought to be lost can, under certain circumstances, be recovered from hard disks using digital forensic methods, which can be costly and time-consuming. Even when digital data is retained, it is malleable and subject to undetected alterations of content or metadata. Lack of proper data retention processes makes it more difficult, sometimes impossible, to recover original data files and verify their integrity.\nGenerally, normal backup processes do not have the fidelity of digital forensic preservation mechanisms. To manage the risks of data loss and undetected alterations, traditional data retention practices in forensic laboratories can be updated to employ digital forensic preservation methods. Specifically, as part of routine data retention processes, digital forensic preservation of original data (raw and processed) and associated metadata (filesystem timestamps) allows the integrity of data to be verified more easily when there is a problem or inquiry. For instance, original files and associated metadata can be forensically preserved using the Advanced Forensic Format (AFF4), which is open-source and cross-platform. The following command and resulting output demonstrate how this method can be implemented on any type of computer system with a single command that can be part of a routine or automated process to forensically preserve all raw data files in a specified directory on a laboratory computer, while generating a unique identifier for the digital evidence container for evidence management purposes[4][5]:\n% aff4.py -cr s1-001-10April2020.aff4 RAWdata\/s1-001\nCreating AFF4Container: file:\/\/s1-001-10April2020.aff4\n<aff4:\/\/c293153c-a317-4927-b1eb-6e3a5008ad0f>\nAdding: RAWdata\nAdding: RAWdata\/s1-001\/s1-001-sequence.sld\nAdding: RAWdata\/s1-001\/s1-001-processed.pdf\nAdding: RAWdata\/s1-001\/s1-001-ref.params\nAdding: RAWdata\/s1-001\/s1-001.RAW<\/tt>\nThis digital forensic preservation process captures file system metadata and automatically computes MD5 and SHA1 cryptographic hash values of the acquired data for integrity verification purposes as the following excerpt shows:\n% aff4.py -m s1-000-10April2020.aff4\n... EDITED FOR BREVITY...\n<aff4:\/\/c293153c-a317-4927-b1eb-6e3a5008ad0f\/RAWdata\/s1-001\/s1-000.RAW>\na aff4:FileImage,\naff4:Image,\naff4:ImageStream;\naff4:birthTime \u201c2020-04-10T22:41:03.949269+02:00\u201d^\u25efsd:dateTime;\naff4:hash \u201c1d2f7ff1ea563ceb6d2da0e168e90587\u201d^\u00e2ff4:MD5,\n\u201c427bc17e608fc493f0e2b3fed8fa55b36862ac31\u201d^\u00e2ff4:SHA1;\naff4:lastAccessed \u201c2020-04-10T22:41:08.708498+02:00\u201d^\u25efsd:dateTime;\naff4:lastWritten \u201c2020-04-10T22:41:05.290019+02:00\u201d^\u25efsd:dateTime;\naff4:originalFileName \u201cRAWdata\/s1-001\/s1-000.RAW\u201d^\u25efsd:string;\naff4:recordChanged \u201c2020-04-10T22:41:07.694584+02:00\u201d^\u25efsd:dateTime;\naff4:size 276196936.<\/tt>\nThese hash values are commonly used in digital forensic tools to enable future verification that the acquired data have not been altered since they were forensically preserved. The preserved metadata can also be useful for assessing the authenticity of the acquired data, including the original file name, size and creation timestamp.\nAdditionally, AFF4 assigns a unique identifier to the acquired data to support evidence management and provenance tracking.\n\nEvidence integrity \nThe data files generated by laboratory equipment and stored on computers can be altered afterwards accidentally or intentionally.\n\nData alteration scenario \nIn this scenario, imagine data files stored on laboratory computers have been altered to conceal specific information in test results. Some alterations were detectable within the digital file, while others were not detected using available verification software. As a result, it was difficult to determine the full scope and specific impact of the alterations.\nThe motivation for editing the data files (raw and processed) might be to cover up mistakes, conceal unfavorable results (corruption), facilitate prosecution (bias), or inflate laboratory metrics (performance)[6] Forensic laboratory personnel might modify data to remove traces of contamination they considered to be insignificant, such as traces of investigators operating an evidential smartphone after the device was seized. Depending on the type of data and the method of modification, it might be possible to detect the alteration. However, some alterations may be undetectable using existing verification tools, making it more difficult to determine that modifications were made.\nNormal backup processes, and even digital forensic preservation such as described in the previous section using AFF4, are not tamperproof because data can be forged to replace retained data, and a computer system can be backdated to make it seem to have occurred sometime in the past. Lack of a tamperproof chain of custody of primary data sources in a forensic laboratory makes it more difficult, sometimes impossible, to authenticate original data files that form the basis of forensic findings and reported results.\nTo manage the risks of inadvertent alteration and intentional tampering, traditional provenance tracking practices in forensic laboratories must be updated to employ digitalized chain of custody ledger solutions.[7][8] These digitalized chain of custody mechanisms can be implemented in a way that is tamperproof and independently verifiable.\n\nData traceability \nForensic laboratories are increasingly using a LIMS to record information about the full lifecycle of evidence in a forensic laboratory, including submission data, chain of custody, and results. A typical LIMS uses databases to store and organize information about each item of evidence at different stages of its treatment in the laboratory.\nA LIMS is invaluable for keeping track of the growing amount of evidence and associated processes and results in forensic laboratories. As a result, such systems are considered essential for laboratory accreditation under standards such as ISO\/IEC 17025. However, these systems can have weaknesses, including user-based data entry errors, programming bugs, and system administrator bypass of access controls.\n\nLIMS weaknesses scenario \nIn this scenario, imagine that the results of drug tests have been routinely recorded in a LIMS, and normal users of the system can only create new records and view existing records. However, a system administrator was able to alter records using his higher level access, bypassing the security control mechanisms of a LIMS. As such, the LIMS maintained an audit log of all normal user activities, but did not log administrator-level actions.\nTo manage these risks of undetected or unattributed alterations to LIMS data, it is necessary to require unique user accounts for all actions and to maintain detailed electronic audit logs. These audit logs must include successful actions, not only failed or blocked actions. Specifically, all transactions must be recorded (additions, alterations, deletions), and all computer system usage, such as logons and executed commands. In particular, system administrator accounts should be strictly protected (e.g., via two-factor authentication) and monitored (e.g., via sudo logging and process accounting). All audit logs must be preserved in a forensically sound manner in anticipation of their use as digital evidence in a legal matter. Applying digital forensic preservation and digitalized chain of custody on logs generated by a LIMS and other supporting computer systems can be an efficient way to enhance LIMS traceability.\n\nComputer system malfunction \nForensic laboratories increasingly depend on computers to operate equipment for extracting information from biological and chemical samples (Figure 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Generalized depiction of laboratory equipment creating digital output.\n\n\n\nThe computer systems used to operate laboratory equipment can malfunction, introducing errors in forensic analysis.\n\nHardware issues scenario \nIn this scenario, imagine, unbeknownst to administrators, a few DNA analysis systems in a forensic laboratory are operated by computers with slightly different hardware than the standard configuration. This seemingly minor difference inevitably caused read errors, which resulted in erroneous reference data being accessed on the DNA analysis systems. As a result, incorrect reference data were used in some cases, and the forensic analysis had to be repeated. This demonstrates that a seemingly unrelated problem with computer used to operate equipment for performing laboratory processes can cause incorrect results.\nThis exemplifies how seemingly minor changes to underlying computer systems can interfere with traditional forensic processes. Although validation of computer systems can be covered under existing laboratory management processes, the subtleties of computer hardware and software configurations and interactions must not be underestimated.\n\nAutomation complexity and pitfalls \nIn forensic contexts, use of automated systems, including those with AI and machine learning (ML), support analysis performed by human specialists who interpret the results. Although such automation can help maintain consistency and increase efficiency in forensic analysis, there are several major limitations that must be guarded against. Automated systems can have bugs that produce incorrect results, which can have serious consequences in a forensic context.[9] Additionally, automated AI\/ML systems can introduce bias due to poorly selected training datasets, and can lead to misinterpretations when the results are not fully understood.[10] When automated AI\/ML systems are used to support investigation and forensic analysis, such as comparison of faces in digital video or photographs, algorithmic false positives can lead to incorrect results.\n\nReliability and human rights concerns scenario \nIn this scenario, an independent study of six facial recognition technology test deployments performed by the London Metropolitan Police Service (MPS) found a high number of false positives. In total, only eight out of 46 automatically generated potential face recognition \u201cmatches\u201d that were considered and evaluated by a human were deemed to be correct. The eight verified correct matches were determined by some form of confirmation such as through a street-based identity check.[11] The study paid particular attention to the risks of such technology interfering with fundamental human rights, including privacy violations and discrimination due to algorithmic bias relating to sex, race, or ethnicity.\nThis study highlights the fact that any automated system can have some false positives and false negatives, which demands that human analysts are in the loop to manage the risks of something important being overlooked or misinterpreted. Good practice here can also help deal with the risk of contextual bias. An automated system supporting forensic analysis should offer sequential unmasking capabilities.[12]\nUltimately, automated systems supporting forensic analysis should integrate forensic requirements, risk management, and privacy, i.e., forensic-by-design. If a system is forensic-by-design, forensic preparedness principles and practices have been integrated into the system engineering lifecycle, including risk management, incident handling, forensic principles, and legal compliance.[13] However, such forensic-by-design is not common practice at the moment, and there are growing concerns about the unreliability and invasiveness of such technologies for criminal investigation purposes. Some cities in California are strictly controlling the acquisition and utilization of \"any software, electronic device, system utilizing an electronic device, or similar device used, designed, or primarily intended to collect, retain, process, or share audio, electronic, visual, location, thermal, biometric, olfactory or similar information specifically associated with, or capable of being associated with, any individual or group.\"[14]\nForensic laboratories must be prepared to address concerns about reliability, privacy, and discrimination associated with use of technology for processing data, particularly AI\/ML systems for data analysis.\n\nDigital reinforcement of forensic science principles \nCore principles and processes in forensic laboratories can be bolstered using technology while increasing the efficiency and quality of services. These core principles include authenticity and integrity, reliability and reproducibility, and quality and efficiency. In addition, intelligent application of digital technology can create new opportunities to deal with crime in digitalized society.\n\nAuthenticity and integrity \nDigital transformation of forensic laboratories can enhance the traceability of traces. Existing computerized systems for processing physical and digital evidence typically provide a digital audit trail and its treatment throughout its lifecycle in the forensic laboratory. Alterations to data files (raw and processed) might be detectable using the digital traces that are created routinely during the processing of exhibits in a forensic laboratory. In some situations, to authenticate data files and associated results, it is necessary to perform contextual analysis, including digital forensic analysis of the original data and metadata.\n\nContextual reconstruction scenario \nIn this scenario, questions have arisen about the results of some samples processed by a laboratory. In order to assess the reported results, the raw data was recovered from analysis systems along with processed results. The information from these files was compared with details recorded in the LIMS, and discrepancies were found. However, audit logs in the LIMS were found that corresponded with the original data. Timestamps of the original data files and the LIMS were also examined to determine contemporaneous activities versus later changes.\nAs this scenario highlights, it is advisable not to leave evidence authentication to chance. With properly protected audit trail and data file integrity mechanisms (preferably automated), a digitalized chain of custody can be maintained from evidence intake to evidence return or archiving, providing valuable insights into laboratory operations and helping protect against undetected mistakes and forensic fraud. Some forensic laboratories have adopted bar code scanning and RFID labels to mark and track exhibits throughout their lifecycle. Some forensic laboratories use secure storage to retain raw data files. Some laboratories are developing enhanced methods for maintaining provenance information using blockchain-based systems.[7][8]\nWith such automated provenance tracking mechanisms in place, every deliverable that a forensic laboratory produces could include the associated electronic chain of custody details as an appendix to demonstrate proper handling. Providing this provenance information in a standardized format such as CASE enables receiving organizations to integrate the information into their information management systems and detect potential inconsistencies more easily, and even automatically.\n\nReliability and reproducibility \nForensic science practices demand that the data, methods, tools, and analysis are described in sufficient detail that they can be carried out again with the same results.\nDigital transformations can support reproducibility of processing by documenting all phases of the evidence lifecycle in the forensic laboratory, as well as reproducibility of analysis by providing others with original data to perform independent analysis. Some software developers also provide a standalone application for viewing the results of forensic processes, enabling others to verify forensic analysis without requiring them to purchase licenses for the specific software and\/or version.\n\nReproducibility of forensic analysis scenario \nIn this scenario, multiple parties need to perform an in-depth review of the results produced by a forensic laboratory, as part of an investigation involving various computers and mobile devices. To save cost and time, the forensic laboratory provided the multiple parties with a complete package of data and viewing software necessary to perform in-depth review. Using this approach, all parties could assess the reliability of the evidence and results without having to purchase specialized equipment and software, or to run time consuming data processing operations that had already been performed by the laboratory.\nThis scenarion shows that expanding decentralisation of forensic capabilities is driving the need for data and analysis results to be transferred securely between systems and organizations that is fully traceable, if not completely repeatable. This trend raises the importance of international standards for harmonization of forensic methods and data formats.\n\nQuality and efficiency \nTechnological advances enable forensic laboratories to process evidence more quickly while maintaining quality. Forensic laboratories can manage the systems that they rely on for processing evidence in order to standardize the processing and output. The National Institute of Standards and Technology (NIST) Computer Forensic Tool Testing (CFTT) program publishes open access test reports for software commonly used in digital forensic laboratories, detailing both capabilities and limitations. The European Network of Forensic Science Institutes (ENFSI) and the US DoD Cyber Crime Center (DC3) also perform tool validation and testing, and provide the results to law enforcement agencies. Similar validation and testing should be applied to computers and software used to operate traditional forensic equipment. As with the prior discussed scenario of computer system malfunction, such validation must be alert for seemingly insignificant changes that can create major errors in traditional forensic processes. Furthermore, the validation of AI\/ML based software is an important and complex undertaking that must be performed by an independent entity or consortium. This level of control helps improve the consistency and quality of results from forensic laboratory processes.\n\nValidation and change control scenario \nIn this scenario, image that a forensic laboratory routinely validates and tests equipment, and maintains strict change control procedures for computers used to process evidence. Every aspect of the systems gets covered, including the specific hardware, firmware, and software versions. Validation of the forensic processes gets performed on the specific systems, and any issues are recognized and dealt with before causing further problems.\n\nForensic-by-design automation \nForensic laboratories are developing new automated systems, including those with AI\/ML capabilities, to analyze data and to gain understanding of crime in ways that were previously not feasible.[10][15]\n\nForensic artificial intelligence scenario \nIn this scenario, imagine that electronic portable devices, based on near-infrared spectroscopy and supported by machine learning, have been developed to test drugs rapidly. These devices could be useful for producing a full profile of the substance tested in the field and transfer the data to a computerized repository in the laboratory for further analysis.[16] Such a repository could be useful for tracking drug trends in a region, and quickly detecting a problem such as an epidemic.\nThe value of forensic laboratories can be extended to more proactive intelligence-led approaches based on knowledge extracted from repetitions found in crime.[15] To manage the associated risks, systems supporting such intelligence-based solutions should be forensic-by-design, abiding by forensic principles and human rights, including privacy and nondiscrimination. Forensic-by-design includes traceability, encryption, and impossibility even for system administrators to access private information.\n\nDigital transformation risk management \nThis section describes a series of recommendations (Table 1) for managing risks of digital transformation of forensic laboratories. Forensic laboratories need to prepare for matters such as lost evidence, being audited or investigated, and responding to civil lawsuits. In the digital forensic domain, the practice of forensic preparedness has been developed to manage risks associated with computer use and misuse. The practice of forensic preparedness involves specification of a policy that lays down a consistent approach, detailed planning against typical (and actual) audit or investigative scenarios that an organization faces, identification of (internal or external) resources that can be deployed as part of those plans, identification of where and how the associated digital evidence can be gathered that will support investigation, and a process of continuous improvement that learns from experience.[17]\n\n\n\n\n\n\n\nTable 1. Road-map for digital transformation risk management\n\n\nRisk\n\nRecommendation\n\nAssociated subsection (below)\n\n\nNot knowing what data exist\n\nDevelop a digital evidence map documenting where needed data, logs, and provenance details are located and how they will be preserved in a forensic manner.\n\n\"Curated digital information\"\n\n\nNot having a plan\n\nInstitute policies and procedures that govern the roles, responsibilities, and expected actions during an inquiry.\n\n\"Established processes and procedures\"\n\n\nMissing data and metadata\n\nImplement an automated digital forensic preservation process of primary data sources.\n\n\"Data integrity\"\n\n\nLack of traceability\n\nMaintain and forensically preserve detailed electronic audit logs of all computer system usage and database transactions.\n\n\"Strategic audit logging\"\n\n\nLack of data integrity\n\nTamperproof the digitalized chain of custody ledger to support authentication of data files and audit logs.\n\n\"Data authentication\"\n\n\nLack of practice\n\nUse audit logs and digitalized chain of custody records for routine purposes to ensure that the logs are monitored and used regularly.\n\n\"Regular use or review of data and processes\"\n\n\nInsufficient validation\n\nValidate computer-reliant forensic processes whenever there are changes to hardware or software configurations.\n\n\"Test and validation\"\n\n\nBlack-box automation\n\nEvaluate automated systems at three levels: performance, understandability, and scientific interpretation.\n\n\"Effective use of automation\"\n\n\nLack of digital forensic support\n\nEngage digital forensic expertise in preparation for problems, and to assist with dealing with problems that arise in a forensic laboratory.\n\n\"Role of digital forensic expertise\"\n\n\n\nLack of forensic preparedness increases the risks of problems going undetected and of ineffective response after a problem is detected. A reactive approach is costly and disruptive, including the need to find and retain external digital forensic expertise. Forensic preparedness enhances business continuity and contingency planning, putting organizations in a better position to detect and investigate problems and manage the associated risks.[18][19] These issues and remedies apply equally to all forensic laboratories, including those within private enterprises.\nIt is important to note that this road map of recommendations does not cover cyberattacks, which require additional digital forensic preparations such as incident response procedures and expertise.\n\nCurated digital information \nA fundamental aspects of being prepared from a digital forensic perspective is knowing where key data sources are located, and ensuring that they contain the minimum data necessary to support business needs and meet legal requirements.\nThe purpose of curating digital information is to reduce the amount of time required to access and examine relevant data, thus reducing the interruption of business continuity and the overall cost to the forensic laboratory. Minimizing interruptions in forensic laboratories is essential for criminal justice, and some countries require annual testing of business continuity plans, with guidance from ISO 22313:2012 Societal security \u2014 Business continuity management systems \u2014 Guidance.[20]\nCurating digital information entails identifying all computer-reliant processes and then augmenting data sources to facilitate authentication of digital evidence (Figure 2). When curating digital information, it is also important to eliminate extraneous data in order to avoid inordinately large repositories that are costly and time-consuming to search or produce.[21] \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Example digital evidence map for forensic laboratories.\n\n\n\nForensic laboratories cannot rely solely on system administrators to curate digital information sources because these individuals are primarily concerned with system performance and security, and they have limited experience with forensic preparedness. To ensure that forensic preparation will withstand scrutiny, it is advisable to involve experienced digital forensic practitioners in the process.\nAn important aspect of this forensic preservation process is to have a data management policy that specifies how long data will be maintained. For instance, a data management strategy could include a control to erase the original files after they are forensically preserved, to avoid having to perform a costly and time-consuming forensic preservation and examination of every computer in the laboratory whenever there is a problem. Data retention decisions must take into account applicable national laws and regulations of the country.\nOne of the most useful tools that investigators can have is a map indicating where evidence is located on a network, i.e., a digital evidence map. Such a map is even more useful when it specifies how long digital evidence remains on the network and references procedures for collecting the evidence.[22]\n\nEstablished processes and procedures \nHaving documented processes to handle common audit and investigative scenarios puts forensic laboratories in a stronger position to respond in an organised, efficient, and effective manner. Effective forensic preparedness requires clear oversight, authorizations, responsibilities, expected actions, desired results, and restrictions. For example, who will gather needed data from various sources in the forensic laboratory (e.g., from the LIMS, computers, backups), how they will perform these tasks, and under whose authority of supervision.\nThe purpose of documented processes and procedures is to ensure a more organized and efficient response and to reduce the risk of mistakes and oversights, thus limiting the associated disruption, liability, and cost for forensic laboratories.\nThese procedures include preservation of digital data with minimal disruption to business continuity and forensic laboratory operations. Proper preservation of digital data supports investigative, forensic, legal, and regulatory requirements, and can help a forensic laboratory defend against any subsequent civil litigation. Preservation is vastly simplified when forensic preparations are already in place.\nForensic laboratories should periodically test these processes and procedures to ensure they work as expected.\n\nData integrity \nManufacturers of laboratory equipment focus on the intended use of their systems, rather than potential misuses. Although manufacturers have some ability to check that output data is well formed, they cannot detect all types of alteration or corruption. Therefore, forensic laboratories cannot rely solely on the manufacturers of equipment to detect alteration or corruption of their output data. A straightforward mechanism for assuring the integrity of data is to generate cryptographic hash values of files, as demonstrated in the previously discussed data retention and loss scenario. Using a digital forensic preservation method such as AFF4 provides more robust data integrity and evidence management. The integrity information can be stored in a LIMS system along with other pertinent details for an evidence item, or in a tamperproof digitalized chain of custody ledger.\n\nStrategic audit logging \nForensic laboratories need to pay careful attention to maintaining chain of custody, including generating and protecting audit logging and mechanisms to ensure the authenticity and integrity of data in a LIMS, as well as generated instrument files (raw and processed).\nReady access to audit logs allows faster response to problems and more comprehensive assessment of the scope of a problem, thereby helping reduce the associated interruption and cost. When a problem occurs in a forensic laboratory, it is advisable to make a forensic copy of all audit logs, preserving them as a source of evidence for further analysis. In some situations, it might also be necessary to restore older audit logs from backup to obtain a complete view of relevant activities.\nUnder certain conditions, system administrator level access can be used to bypass many protections and access controls on computer systems. Therefore, additional measures are needed to maintain the integrity of data in forensic laboratories such as maintaining integrity records contemporaneously, as discussed in the next section.\n\nData authentication \nTo enable authentication of data, forensic results, and audit logs, some forensic laboratories print hard copies of certain data to maintain paper records for future comparison with digital information. Another approach is to archive copies of the original digital data on read-only storage media. Authentication can be further enhanced with a tamper-proof digitalized chain of custody ledger using blockchain infrastructure, as referenced above.\nGiven the importance of temporal information for data authentication, it is important for clocks on all computer systems to be automatically synchronized with a single timesource and timezone.\nSome laboratories are adopting the CASE specification to represent digital forensic information, including provenance details, in a standardised form, maintaining provenance of exhibits throughout their lifecycle in forensic laboratories.[23] Data marking is an integral part of CASE, supporting data protection for privacy and security purposes.[24]\n\nRegular use or review of data and processes \nA common mistake that organizations make is to configure data retention and logging and only examine the information after a problem occurs. As a result, problems are registered in preserved data and audit logs but are not observed by anyone in the organization.\nRoutinely relying on the curated data sources, or at least reviewing them to ensure that those responsible are familiar with using and interpreting them, have effective tools for examining them, and promptly notice and resolve failures or errors such as malfunctioning data retention processes, corrupt data, incomplete records, and incorrect time stamps.[25]\n\nTest and validation \nWhen deploying new automated capabilities to support forensic analysis, it is necessary to test and validate the system to determine its reliability and limitations. Finding and mitigating problems in such systems is non-trivial and requires a systematic approach and specialized expertise.[26] Forensic laboratories have the necessary knowledge and structure to manage quality and risk of complex processes, which can be extended to encompass automated AI\/ML systems.[15]\n\nEffective use of automation \nOne way to mitigate the risks associated with automation to support forensic analysis is to realize the value of human expertise. A study of facial comparison found that the optimal results were obtained when forensic expertise was combined with state-of-the-art face recognition technology.[27]\nHowever, when automated systems are not designed with core forensic principles in mind, lacking transparency, there is a risk of the black-box effect.[10] To manage this risk, forensic laboratories need to ensure that all automated processes they rely on produce forensic results, including AI\/ML based systems, can be explained by the laboratory specialists. Limitations of automated systems, including possible false-positives and false-negatives, should be documented to manage risks of errors, omissions, and misuse.\nTo address the risk of incorrect decisions based on automated systems, it is necessary for such systems to function well at three levels: performance, understandability, and scientific interpretation.[28] It is important to address these issues as part of forensic preparedness, before a problem occurs. Such preparations will put forensic laboratories in a stronger position to defend decisions based on an automatic system and explain the underlying logic.\n\nRole of digital forensic expertise \nSuccess in managing risks associated with digital transformations in forensic laboratories depends on the qualifications and experience of the personnel performing the digital forensic processes.\nAlthough there are ongoing efforts to harmonize digital evidence and forensic science[1], some forensic laboratories are currently unable to integrate digital forensic science, due to division of responsibilities or unavailability of resources and expertise. To help manage the risks associated with digital transformations, forensic laboratories can engage external digital forensic expertise to help develop forensic digital preparedness and respond to problems. It is most effective to arrange this prior to a problem occurring.\nWhen a problem occurs, it can be larger than the organization initially realizes. Forensic preparedness makes it easier to determine the full scope of the problem and manage the potential damage more effectively. However, even with perfect preparation, there are usually unexpected challenges in any digital investigation. When the people dealing with a problem are not properly trained in digital forensic science, there is an increased risk of misunderstanding and misinterpretation. Even when they perform their work impeccably, unclear explanation can cause decision makers to miscomprehend digital forensic conclusions.[29] The knowledge and experience required to effectively analyze and evaluate digital forensic evidence should not be underestimated.[30]\nFurthermore, digital forensic analysis can reveal weaknesses in forensic laboratory operations, and digital forensic expertise can help continuously improve forensic processes.[30]\n\nQuality assurance considerations \nThe forensic preparedness measures discussed above not only help laboratories respond more effectively and efficiently to problems, they also help improve quality assurance and auditability. A digital evidence map, digitally strengthened chain of custody ledger, and strategic audit logging provide more detail and transparency into the handling of exhibits in forensic laboratories, and enables more effective problem detection and response.\nIt is also inadvisable to use the forensic laboratory LIMS to manage evidence of an investigation into its own problems. Therefore, forensic laboratories need to have a separate system for maintaining documentation, incident details, and evidence provenance, and other information that needs to be recorded during investigation of an incident.\nThis raises the question of whether there is a need to incorporate the digital transformation risk mitigation measures discussed in this paper into standards for accreditation such as ISO\/IEC 17025:2017\nGeneral requirements for the competence of testing and calibration laboratories.\n\nCurrent requirements \nValidation within forensic laboratories must include the underlying technology, including their hardware and software configuration, access control (security), and auditability. Each time there is a significant change in hardware or software, routine tests and validation should be performed against known datasets to determine whether expected results are produced.\nNevertheless, according to the authors\u2019 experiences with laboratories both in Europe and the U.S., even ISO 17025 accredited laboratories do not always perform a validation process for each major release of every software application used for forensic analysis. This lack of rigour can be explained by the difficulty to trigger significant changes but is most of the time due to time-saving reasons.\nMoreover, if the ISO 17025 standard requires information management systems to be validated, the current version still states that \u201ccommercial off-the-shelf software in general use within its designed application range can be considered to be sufficiently validated.\u201d[31] This large exception raises questions about the existence and quality of validation processes performed by system providers. Some laboratories might also consider only validating internal systems.\n\nPossible improvements \nTo overcome the limits related to validation cost of commercial software, independent bodies could share their validation results, following the digital forensic tool testing approach taken by NIST and DC3.\nThe need for such validation and testing does not only apply to systems used to process exhibits. More attention could be given to validation of LIMS systems, including their reliability, security, and auditability. Forensic laboratories should also ensure that access controls are in place on LIMS and secure storage systems to prevent unauthorized alteration or deletion. This issue could be address with clearer wording in quality standards such as ISO 17025, or in the the international guidance document ILAC G19.\nThe risks of oversights and misinterpretations are growing as more complex automated systems being used for forensic purposes lack forensics-by-design. To mitigate the associated risks, forensic laboratories need to implement additional measures that govern proper operation and use of automated systems.\nQuality standards such as ISO 17025 could be further refined by adding specific requirements for forensic digital preparedness described in this paper.\n\nConclusions \nForensic laboratories that fail to seriously confront digital transformation risk management will suffer significant disruption and expense when problems arise. To mitigate these risks, forensic laboratories must strengthen their forensic digital preparedness and ensure that technology abides by core principles and processes, i.e., authenticity and integrity, reliability and reproducibility, and quality and efficiency. Laboratories should also consider applying forensic digital preparedness to their email and other administrative systems. As much as feasible, the recommendations in this paper are \u201cenable and forget\u201d (until a problem occurs). An initial investment in forensic preparation can return repeated benefits by reinforcing forensic principles, as noted in the section \"Digital reinforcement of forensic science principles,\" and by reducing expenses and business disruption each time a problem arises.\nIn brief, laboratories should take forensic digital preparedness steps before a problem arises, and consider involving digital forensic specialists with experience in forensic laboratory operations. When a problem occurs, follow documented processes and procedures for responding to such incidents. If a plan does not exist, create one and implement it methodically. Know where sources of relevant digital data are, and take steps to preserve them properly. Forensic preservation of data includes original files and backups, as well as audit logs. Perform a thorough scope assessment to determine the full extent of the problem. Document all actions taken in response to the problem.\nIt would be generally beneficial to require LIMS and other digitalized processes in forensic laboratories to be forensic-by-design. The recommendations in this paper provide a foundation for forensic laboratories to develop requirements that system providers should fulfill.\nWith proper forethought and preparation, forensic laboratories can employ technology and advanced data analytics to enhance existing services and create new services, while respecting fundamental human rights.\nApplying their existing knowledge and structures, forensic laboratories are in a strong position to effectively manage quality and risk of digital transformations.\n\nAcknowledgements \nOur deepest gratitude to Christophe Champod, Timothy Boll\u00e9, Francesco Servida, and Hannes Spichiger for their collaboration and insights. We are also thankful for the peer reviewers' recommendations, which improved this work.\n\nAuthor contributions \nConception and design of study: Eoghan Casey, Thomas Souvignet. Drafting the manuscript: Eoghan Casey, Thomas Souvignet. Approval of the version of the manuscript to be published: Eoghan Casey, Thomas Souvignet.\n\nConflicts of interest \nNone stated.\n\nReferences \n\n\n\u2191 1.0 1.1 Pollitt, M.; Casey, E.; Jaquet-Chiffelle, D.-O. et al. (February 2019). \"A Framework for Harmonizing Forensic Science Practices and Digital\/Multimedia Evidence\" (PDF). OSAC. https:\/\/www.nist.gov\/system\/files\/documents\/2018\/01\/10\/osac_ts_0002.pdf .   \n\n\u2191 Tully, G.; Cohen, N.; Compton, D. et al. (2020). \"Quality standards for digital forensics: Learning from experience in England & Wales\". Forensic Science International: Digital Investigation 32: 200905. doi:10.1016\/j.fsidi.2020.200905.   \n\n\u2191 Reust, J.; Sommers, R.; Friedberg, S. et al. (2008). \"Identification and Reconstruction of Deleted, Fragmented DNA Digital Files\". Proceedings of the American Academy of Forensic Sciences 14: 187\u201388. Archived from the original on 29 April 2016. https:\/\/web.archive.org\/web\/20160429004454\/https:\/\/www.aafs.org\/wp-content\/uploads\/ProceedingsWashingtonDC2008.pdf .   \n\n\u2191 Cohen, M.; Garfinkel, S.; Schatz, B. (2009). \"Extending the advanced forensic format to accommodate multiple data sources, logical evidence, arbitrary information and forensic workflow\". Digital Investigation 6 (Supplement 1): S57\u2013S68. doi:10.1016\/j.diin.2009.06.010.   \n\n\u2191 Schatz, B.L. (2015). \"Wirespeed: Extending the AFF4 forensic container format for scalable acquisition and live analysis\". Digital Investigation 14 (Supplement 1): S45\u201354. doi:10.1016\/j.diin.2015.05.016.   \n\n\u2191 Bidgood, J. (18 April 2017). \"Chemist\u2019s Misconduct Is Likely to Void 20,000 Massachusetts Drug Cases\". The New York Times. Archived from the original on 19 April 2017. https:\/\/web.archive.org\/web\/20170419100845\/https:\/\/www.nytimes.com\/2017\/04\/18\/us\/chemist-drug-cases-dismissal.html .   \n\n\u2191 7.0 7.1 Burri, X.; Casey, E.; Boll\u00e9, T. et al. (2020). \"Chronological independently verifiable electronic chain of custody ledger using blockchain technology\". Forensic Science International: Digital Investigation 33: 300976. doi:10.1016\/j.fsidi.2020.300976.   \n\n\u2191 8.0 8.1 Jaquet-Chiffelle, D.-O.; Casey, E.; Bourquenoud, J. (2020). \"Tamperproof timestamped provenance ledger using blockchain technology\". Forensic Science International: Digital Investigation 33: 300977. doi:10.1016\/j.fsidi.2020.300977.   \n\n\u2191 Murray, D. (20 March 2015). \"Queensland authorities confirm \u2018miscode\u2019 affects DNA evidence in criminal cases\". The Courier Mail. https:\/\/www.couriermail.com.au\/news\/queensland\/queensland-authorities-confirm-miscode-affects-dna-evidence-in-criminal-cases\/news-story\/833c580d3f1c59039efd1a2ef55af92b .   \n\n\u2191 10.0 10.1 10.2 Margagliotti, G.; Boll\u00e9, T. (2019). \"Machine learning & forensic science\". Forensic Science International 298: 138\u201339. doi:10.1016\/j.forsciint.2019.02.045.   \n\n\u2191 Fussey, P.; Murray, D. (2019). \"Independent Report on the London Metropolitan Police Service\u2019s Trial of Live Facial Recognition Technology\". University of Essex Human Rights Centre. http:\/\/repository.essex.ac.uk\/24946\/ .   \n\n\u2191 Koppl, R. (2019). \"Strategic choice in linear sequential unmasking\". Science & Justice 59 (2): 166-171. doi:10.1016\/j.scijus.2018.10.010.   \n\n\u2191 Rahman, N.H.A.; Glisson, W.B.; Yang, Y. et al. (2016). \"Forensic-by-Design Framework for Cyber-Physical Cloud Systems\". IEEE Cloud Computing 3 (1): 50\u201359. doi:10.1109\/MCC.2016.5.   \n\n\u2191 \"Administrative Code - Acquisition of Surveillance Technology, Ordinance No. 107-19\" (PDF). County of San Francisco. 21 May 2019. https:\/\/sfbos.org\/sites\/default\/files\/o0107-19.pdf .   \n\n\u2191 15.0 15.1 15.2 Casey, E.; Ribaux, O.; Roux, C. (2019). \"The Kodak Syndrome: Risks and Opportunities Created by Decentralization of Forensic Capabilities\". Journal of Forensic Sciences 64 (1): 127\u201336. doi:10.1111\/1556-4029.13849.   \n\n\u2191 Coppey, F.; B\u00e9cue, A.; Sacr\u00e9, P.-Y. et al. (2020). \"Providing illicit drugs results in five seconds using ultra-portable NIR technology: An opportunity for forensic laboratories to cope with the trend toward the decentralization of forensic capabilities\". Forensic Science International 317: 110498. doi:10.1016\/j.forsciint.2020.110498.   \n\n\u2191 CESG (2015). \"Forensic Readiness\". Good Practice Guide (1.2). https:\/\/vdocuments.mx\/download\/good-practice-guide-forensic-readiness-ncsc-site-18aa-forensic-readiness .   \n\n\u2191 Johnston, A.; Reust, J. (2006). \"Network intrusion investigation \u2013 Preparation and challenges\". Digital Investigation 3 (3): 118\u201326. doi:10.1016\/j.diin.2006.08.001.   \n\n\u2191 Elyas, M.; Ahmad, A.; Maynard, S.B. et al. (2015). \"Digital forensic readiness: Expert perspectives on a theoretical framework\". Computers & Security 52: 70\u201389. doi:10.1016\/j.cose.2015.04.003.   \n\n\u2191 Tully, G. (05 September 2014). \"Forensic science providers: codes of practice and conduct\". Gov.UK. UK Government. https:\/\/www.gov.uk\/government\/collections\/forensic-science-providers-codes-of-practice-and-conduct .   \n\n\u2191 Casey, E. (2007). \"Digital evidence maps \u2013 A sign of the times\". Digital Investigation 4 (1): 1\u20132. doi:10.1016\/j.diin.2007.01.004.   \n\n\u2191 Casey, E. (2011). Digital Evidence and Computer Crime: Forensic Science, Computers, and the Internet. Academic Press, Inc. pp. 840. ISBN 9780123742681.   \n\n\u2191 Casey, E.; Barnum, S.; Griffith, R. et al. (2017). \"Advancing coordinated cyber-investigations and tool interoperability using a community developed specification language\". Digital Investigation 22: 14\u201345. doi:10.1016\/j.diin.2017.08.002.   \n\n\u2191 Casey, E.; Biasiotti, M.A.; Turchi, F. (2017). \"Using Standardization and Ontology to Enhance Data Protection and Intelligent Analysis of Electronic Evidence\" (PDF). https:\/\/serval.unil.ch\/resource\/serval:BIB_EFAFD05944CB.P001\/REF.pdf .   \n\n\u2191 Casey, E.; Daywalt, C.; Johnston, A. (2010). \"Chapter 4: Intrusion Investigation\". In Casey, E.. Handbook of Digital Forensics and Investigation. Academic Press, Inc. pp. 135\u2013206. doi:10.1016\/B978-0-12-374267-4.00004-5. ISBN 9780123742674.   \n\n\u2191 Taylor, D.A.; Bright, J.-A.; Buckleton, J. (2017). \"Commentary: A \u201cSource\u201d of Error: Computer Code, Criminal Defendants, and the Constitution\". Frontiers in Genetics 8: 33. doi:10.3389\/fgene.2017.00033.   \n\n\u2191 Phillips, P.J.; Yates, A.N.; Hu, Y. et al. (2018). \"Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms\". PNAS 115 (24): 6171\u201376. doi:10.1073\/pnas.1721355115.   \n\n\u2191 Boll\u00e9, T.; Casey, E.; Jacquet, M. et al. (2020). \"The role of evaluations in reaching decisions using automated systems supporting forensic analysis\". Forensic Science International: Digital Investigation 34: 301016. doi:10.1016\/j.fsidi.2020.301016.   \n\n\u2191 Casey, E. (2019). \"Trust in digital evidence\". Digital Investigation 31: 200898. doi:10.1016\/j.fsidi.2019.200898.   \n\n\u2191 30.0 30.1 Casey, E. (2020). \"Standardization of forming and expressing preliminary evaluative opinions on digital evidence\". Forensic Science International: Digital Investigation 32: 200888. doi:10.1016\/j.fsidi.2019.200888.   \n\n\u2191 \"ISO\/IEC 17025:2017 General requirements for the competence of testing and calibration laboratories\". International Organization for Standardization. November 2017. https:\/\/www.iso.org\/standard\/66912.html .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\">https:\/\/www.limswiki.org\/index.php\/Journal:Digital_transformation_risk_management_in_forensic_science_laboratories<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on forensic scienceLIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on risk management\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 30 March 2021, at 17:58.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 304 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","b18b6bb1bbee57c49590e7b3ba53097d_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Digital_transformation_risk_management_in_forensic_science_laboratories skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Digital transformation risk management in forensic science laboratories<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Technological advances are changing how <a href=\"https:\/\/www.limswiki.org\/index.php\/Forensic_laboratory\" title=\"Forensic laboratory\" class=\"mw-redirect wiki-link\" data-key=\"dabef2566b55c1f13b395543b47ae81e\">forensic laboratories<\/a> operate in all <a href=\"https:\/\/www.limswiki.org\/index.php\/Forensic_science\" title=\"Forensic science\" class=\"wiki-link\" data-key=\"415d36a7b65494677b6d2873d5febec1\">forensic disciplines<\/a>, not only digital. Computers support <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a> management and enable evidence analysis (physical and digital), while new technology enables previously unavailable forensic capabilities. Used properly, the integration of digital systems supports greater efficiency and reproducibility, and drives digital transformation of forensic laboratories. However, without the necessary preparations, these digital transformations can undermine the core principles and processes of forensic laboratories. Forensic preparedness concentrating on digital data reduces the cost and operational disruption of responding to various kinds of problems, including misplaced exhibits, allegations of employee misconduct, disclosure requirements, and information security breaches. \n<\/p><p>This work gives pertinent examples of problems and risks involving technology that have occurred in forensic laboratories, along with opportunities and risk mitigation strategies, based on the authors\u2019 experiences. It also presents recommendations to help forensic laboratories prepare for and manage these risks, to use technology effectively, and ultimately strengthen forensic science. The importance of involving digital forensic expertise in risk management of digital transformations in laboratories is emphasized. Forensic laboratories that do not adopt forensic digital preparedness will produce results based on digital data and processes that cannot be verified independently, leaving them vulnerable to challenge. The recommendations in this work could enhance international standards such as <a href=\"https:\/\/www.limswiki.org\/index.php\/ISO\/IEC_17025\" title=\"ISO\/IEC 17025\" class=\"wiki-link\" data-key=\"0a89cebb34370dd860cce86881cbf29c\">ISO\/IEC 17025<\/a>, which are used to assess and accredit laboratories.\n<\/p><p><b>Keywords<\/b>: forensic science, digital transformations, forensic laboratories, forensic preparedness, forensic digital preparedness, risk management, ISO\/IEC 17025, laboratory information management systems\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Forensic_laboratory\" title=\"Forensic laboratory\" class=\"mw-redirect wiki-link\" data-key=\"dabef2566b55c1f13b395543b47ae81e\">Forensic science laboratories<\/a> are becoming more reliant on computers and data for both administrative and analytical operations. These technological advances create new opportunities and risks for all <a href=\"https:\/\/www.limswiki.org\/index.php\/Forensic_science\" title=\"Forensic science\" class=\"wiki-link\" data-key=\"415d36a7b65494677b6d2873d5febec1\">forensic disciplines<\/a>, not only to digital evidence.<sup id=\"rdp-ebb-cite_ref-PollittAFrame19_1-0\" class=\"reference\"><a href=\"#cite_note-PollittAFrame19-1\">[1]<\/a><\/sup> With proper preparation and management, forensic laboratories can employ technology effectively to improve performance and quality, while mitigating the associated risks. However, many forensic laboratories do not understand the subtlety and expertise required to manage risks of digital transformation, inadvisedly treating it as simply a technical component of existing <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_management_system\" title=\"Quality management system\" class=\"wiki-link\" data-key=\"dfecf3cd6f18d4a5e9ac49ca360b447d\">quality management processes<\/a>. Forensic laboratories that fail to realize the need for forensic digital preparedness to actively manage risks associated with digital transformations are vulnerable to significant expense, disruption, and liability when problems arise.\n<\/p><p>Forensic laboratories rely on technology for much more than communication and routine business functions. Sophisticated equipment for processing chemical and biological materials are operated using computers and save results in digital form. <a href=\"https:\/\/www.limswiki.org\/index.php\/Mass_spectrometry\" title=\"Mass spectrometry\" class=\"wiki-link\" data-key=\"fb548eafe2596c35d7ea741849aa83d4\">Mass spectrometers<\/a>, DNA analysis systems, and other <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> equipment save their results in raw data files. Digital evidence is processed using specialized hardware and software, although not all forensic laboratories have integrated this new discipline. Forensic laboratories are using computerized case management systems for tracking treatment of all evidential exhibits and forensic results. Automated systems with <a href=\"https:\/\/www.limswiki.org\/index.php\/Artificial_intelligence\" title=\"Artificial intelligence\" class=\"wiki-link\" data-key=\"0c45a597361ca47e1cd8112af676276e\">artificial intelligence<\/a> (AI) are being used to support forensic analysis. In reality, digital transformations\u2014the use of digital technology to make existing processes more efficient and effective, and to develop new solutions to emerging problems\u2014are well underway, and forensic laboratories require a robust strategy to manage the associated risks and realize the opportunities.\n<\/p><p>This increased dependence on digital technology creates risks and opportunities for forensic laboratories. Potential pitfalls include loss of data needed to perform forensic analysis, errors in <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analysis<\/a> of physical traces (e.g., DNA, fingerprint, face) caused by computer hardware or software, ability to tamper with raw data files generated by laboratory equipment, and incorrect <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> input into <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> (LIMS). Possible benefits are traceability and integrity of traces, reliability and reproducibility of results from information extracted from traces and stored as raw data, and use of AI to support forensic analysis.\n<\/p><p>Lessons can be learned from the digital forensic domain, including forensic digital preparedness and accreditation challenges. Primary challenges encountered by digital forensic laboratories adopting quality standards include<sup id=\"rdp-ebb-cite_ref-TullyQuality20_2-0\" class=\"reference\"><a href=\"#cite_note-TullyQuality20-2\">[2]<\/a><\/sup>:\n<\/p>\n<ul><li> Inaccurate or insufficient information in technical records, including <a href=\"https:\/\/www.limswiki.org\/index.php\/Chain_of_custody\" title=\"Chain of custody\" class=\"wiki-link\" data-key=\"6ba04fe2bb1c8375e133455821aa3894\">chain of custody<\/a>, and no mechanism to detect subsequent changes to records.<\/li>\n<li> Problems with the security of information technology systems and the <a href=\"https:\/\/www.limswiki.org\/index.php\/Backup\" title=\"Backup\" class=\"wiki-link\" data-key=\"e12548e6bf5f28bfee99099fe8662dde\">backup<\/a> processes of data.<\/li>\n<li> Missing or insufficiently detailed procedures for treating digital data, and personnel not following documented procedures consistently.<\/li>\n<li> Lack of robust <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality checking<\/a> mechanisms, and issues with validation of methods.<\/li><\/ul>\n<p>This paper presents risks and opportunities associated with digital transformation of forensic laboratories, providing examples based on the authors\u2019 experiences. Examples have been anonymized, as the intention is to illustrate general lessons learned rather than critique specific laboratories. This work then presents forensic digital preparedness, a set of recommendations to help laboratories navigate risks associated with digital transformations, including mishandled exhibits, allegations of employee misconduct, and disclosure requirements. The role of digital forensic capabilities and expertise in risk management of digital transformations in laboratories is discussed. This work culminates with broader implications for international standards such as <a href=\"https:\/\/www.limswiki.org\/index.php\/ISO\/IEC_17025\" title=\"ISO\/IEC 17025\" class=\"wiki-link\" data-key=\"0a89cebb34370dd860cce86881cbf29c\">ISO\/IEC 17025<\/a>, which are used to assess and accredit laboratories.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Risks_and_remedies\">Risks and remedies<\/span><\/h2>\n<p>Many processes in forensic laboratories have become digitalized through the increased use of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">information management<\/a> systems and software running analysis instruments. While these systems serve crucial functions in modern forensic laboratories, thet also have associated risks that must be managed.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_retention\">Data retention<\/span><\/h3>\n<p>The computer systems used to store instruments' generated data files (raw and processed) can encounter problems that lead to loss of information.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Data_loss_scenario\">Data loss scenario<\/span><\/h4>\n<p>In this scenario, Reust <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ReustIdent08_3-0\" class=\"reference\"><a href=\"#cite_note-ReustIdent08-3\">[3]<\/a><\/sup> presented a case study concerning a forensic laboratory that performed DNA analysis of a crime scene sample relevant to a multiple homicide and death penalty case, but did not retain a copy of the raw data files. To comply with a court order to provide the defense with original raw data, it was necessary to perform costly forensic data recovery on the computer used to perform the original processing of DNA. The authors developed a customized software utility to automatically search the computer hard drive for all fragments of the relevant raw data and reconstruct the original files. The resulting files were tested and validated with DNA analysis software.\n<\/p><p>As seen with Reust <i>et al.<\/i>, original data files thought to be lost can, under certain circumstances, be recovered from hard disks using digital forensic methods, which can be costly and time-consuming. Even when digital data is retained, it is malleable and subject to undetected alterations of content or metadata. Lack of proper data retention processes makes it more difficult, sometimes impossible, to recover original data files and verify their integrity.\n<\/p><p>Generally, normal backup processes do not have the fidelity of digital forensic preservation mechanisms. To manage the risks of data loss and undetected alterations, traditional <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_retention\" title=\"Data retention\" class=\"wiki-link\" data-key=\"d77533b92d003d39cee958a82b62391a\">data retention<\/a> practices in forensic laboratories can be updated to employ digital forensic preservation methods. Specifically, as part of routine data retention processes, digital forensic preservation of original data (raw and processed) and associated metadata (filesystem timestamps) allows the integrity of data to be verified more easily when there is a problem or inquiry. For instance, original files and associated metadata can be forensically preserved using the Advanced Forensic Format (AFF4), which is open-source and cross-platform. The following command and resulting output demonstrate how this method can be implemented on any type of computer system with a single command that can be part of a routine or automated process to forensically preserve all raw data files in a specified directory on a laboratory computer, while generating a unique identifier for the digital evidence container for evidence management purposes<sup id=\"rdp-ebb-cite_ref-CohenExtend09_4-0\" class=\"reference\"><a href=\"#cite_note-CohenExtend09-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SchatzWire15_5-0\" class=\"reference\"><a href=\"#cite_note-SchatzWire15-5\">[5]<\/a><\/sup>:\n<\/p><p><tt>% aff4.py -cr s1-001-10April2020.aff4 RAWdata\/s1-001\n<\/p><p>Creating AFF4Container: file:\/\/s1-001-10April2020.aff4\n<\/p><p><aff4:\/\/c293153c-a317-4927-b1eb-6e3a5008ad0f>\n<\/p><p>Adding: RAWdata\n<\/p><p>Adding: RAWdata\/s1-001\/s1-001-sequence.sld\n<\/p><p>Adding: RAWdata\/s1-001\/s1-001-processed.pdf\n<\/p><p>Adding: RAWdata\/s1-001\/s1-001-ref.params\n<\/p><p>Adding: RAWdata\/s1-001\/s1-001.RAW<\/tt>\n<\/p><p>This digital forensic preservation process captures file system <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a> and automatically computes MD5 and SHA1 hash values of the acquired data for <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">integrity<\/a> verification purposes as the following excerpt shows:\n<\/p><p><tt>% aff4.py -m s1-000-10April2020.aff4\n<\/p><p>... EDITED FOR BREVITY...\n<\/p><p><aff4:\/\/c293153c-a317-4927-b1eb-6e3a5008ad0f\/RAWdata\/s1-001\/s1-000.RAW>\n<\/p><p>a aff4:FileImage,\n<\/p><p>aff4:Image,\n<\/p><p>aff4:ImageStream;\n<\/p><p>aff4:birthTime \u201c2020-04-10T22:41:03.949269+02:00\u201d^\u25efsd:dateTime;\n<\/p><p>aff4:hash \u201c1d2f7ff1ea563ceb6d2da0e168e90587\u201d^\u00e2ff4:MD5,\n<\/p><p>\u201c427bc17e608fc493f0e2b3fed8fa55b36862ac31\u201d^\u00e2ff4:SHA1;\n<\/p><p>aff4:lastAccessed \u201c2020-04-10T22:41:08.708498+02:00\u201d^\u25efsd:dateTime;\n<\/p><p>aff4:lastWritten \u201c2020-04-10T22:41:05.290019+02:00\u201d^\u25efsd:dateTime;\n<\/p><p>aff4:originalFileName \u201cRAWdata\/s1-001\/s1-000.RAW\u201d^\u25efsd:string;\n<\/p><p>aff4:recordChanged \u201c2020-04-10T22:41:07.694584+02:00\u201d^\u25efsd:dateTime;\n<\/p><p>aff4:size 276196936.<\/tt>\n<\/p><p>These hash values are commonly used in digital forensic tools to enable future verification that the acquired data have not been altered since they were forensically preserved. The preserved metadata can also be useful for assessing the authenticity of the acquired data, including the original file name, size and creation timestamp.\n<\/p><p>Additionally, AFF4 assigns a unique identifier to the acquired data to support evidence management and provenance tracking.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Evidence_integrity\">Evidence integrity<\/span><\/h3>\n<p>The data files generated by laboratory equipment and stored on computers can be altered afterwards accidentally or intentionally.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Data_alteration_scenario\">Data alteration scenario<\/span><\/h4>\n<p>In this scenario, imagine data files stored on laboratory computers have been altered to conceal specific information in test results. Some alterations were detectable within the digital file, while others were not detected using available verification software. As a result, it was difficult to determine the full scope and specific impact of the alterations.\n<\/p><p>The motivation for editing the data files (raw and processed) might be to cover up mistakes, conceal unfavorable results (corruption), facilitate prosecution (bias), or inflate laboratory metrics (performance)<sup id=\"rdp-ebb-cite_ref-BidgoodChemist17_6-0\" class=\"reference\"><a href=\"#cite_note-BidgoodChemist17-6\">[6]<\/a><\/sup> Forensic laboratory personnel might modify data to remove traces of contamination they considered to be insignificant, such as traces of investigators operating an evidential smartphone after the device was seized. Depending on the type of data and the method of modification, it might be possible to detect the alteration. However, some alterations may be undetectable using existing verification tools, making it more difficult to determine that modifications were made.\n<\/p><p>Normal backup processes, and even digital forensic preservation such as described in the previous section using AFF4, are not tamperproof because data can be forged to replace retained data, and a computer system can be backdated to make it seem to have occurred sometime in the past. Lack of a tamperproof chain of custody of primary data sources in a forensic laboratory makes it more difficult, sometimes impossible, to authenticate original data files that form the basis of forensic findings and reported results.\n<\/p><p>To manage the risks of inadvertent alteration and intentional tampering, traditional provenance tracking practices in forensic laboratories must be updated to employ digitalized chain of custody ledger solutions.<sup id=\"rdp-ebb-cite_ref-BurriChrono20_7-0\" class=\"reference\"><a href=\"#cite_note-BurriChrono20-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Jaquet-ChiffelleTamper20_8-0\" class=\"reference\"><a href=\"#cite_note-Jaquet-ChiffelleTamper20-8\">[8]<\/a><\/sup> These digitalized chain of custody mechanisms can be implemented in a way that is tamperproof and independently verifiable.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_traceability\">Data traceability<\/span><\/h3>\n<p>Forensic laboratories are increasingly using a LIMS to record information about the full lifecycle of evidence in a forensic laboratory, including submission data, chain of custody, and results. A typical LIMS uses databases to store and organize information about each item of evidence at different stages of its treatment in the laboratory.\n<\/p><p>A LIMS is invaluable for keeping track of the growing amount of evidence and associated processes and results in forensic laboratories. As a result, such systems are considered essential for laboratory accreditation under standards such as ISO\/IEC 17025. However, these systems can have weaknesses, including user-based data entry errors, programming bugs, and system administrator bypass of access controls.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"LIMS_weaknesses_scenario\">LIMS weaknesses scenario<\/span><\/h4>\n<p>In this scenario, imagine that the results of drug tests have been routinely recorded in a LIMS, and normal users of the system can only create new records and view existing records. However, a system administrator was able to alter records using his higher level access, bypassing the security control mechanisms of a LIMS. As such, the LIMS maintained an audit log of all normal user activities, but did not log administrator-level actions.\n<\/p><p>To manage these risks of undetected or unattributed alterations to LIMS data, it is necessary to require unique user accounts for all actions and to maintain detailed electronic audit logs. These audit logs must include successful actions, not only failed or blocked actions. Specifically, all transactions must be recorded (additions, alterations, deletions), and all computer system usage, such as logons and executed commands. In particular, system administrator accounts should be strictly protected (e.g., via two-factor authentication) and monitored (e.g., via sudo logging and process accounting). All audit logs must be preserved in a forensically sound manner in anticipation of their use as digital evidence in a legal matter. Applying digital forensic preservation and digitalized chain of custody on logs generated by a LIMS and other supporting computer systems can be an efficient way to enhance LIMS traceability.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Computer_system_malfunction\">Computer system malfunction<\/span><\/h3>\n<p>Forensic laboratories increasingly depend on computers to operate equipment for extracting information from biological and chemical samples (Figure 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Casey_ForensicSciInt2020_316.jpg\" class=\"image wiki-link\" data-key=\"9b3614e16d02a3976f6b670ed7da37d6\"><img alt=\"Fig1 Casey ForensicSciInt2020 316.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/6f\/Fig1_Casey_ForensicSciInt2020_316.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Generalized depiction of laboratory equipment creating digital output.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The computer systems used to operate laboratory equipment can malfunction, introducing errors in forensic analysis.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Hardware_issues_scenario\">Hardware issues scenario<\/span><\/h4>\n<p>In this scenario, imagine, unbeknownst to administrators, a few DNA analysis systems in a forensic laboratory are operated by computers with slightly different hardware than the standard configuration. This seemingly minor difference inevitably caused read errors, which resulted in erroneous reference data being accessed on the DNA analysis systems. As a result, incorrect reference data were used in some cases, and the forensic analysis had to be repeated. This demonstrates that a seemingly unrelated problem with computer used to operate equipment for performing laboratory processes can cause incorrect results.\n<\/p><p>This exemplifies how seemingly minor changes to underlying computer systems can interfere with traditional forensic processes. Although validation of computer systems can be covered under existing laboratory management processes, the subtleties of computer hardware and software configurations and interactions must not be underestimated.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Automation_complexity_and_pitfalls\">Automation complexity and pitfalls<\/span><\/h3>\n<p>In forensic contexts, use of automated systems, including those with AI and <a href=\"https:\/\/www.limswiki.org\/index.php\/Machine_learning\" title=\"Machine learning\" class=\"wiki-link\" data-key=\"79aab39cfa124c958cd1dbcab3dde122\">machine learning<\/a> (ML), support analysis performed by human specialists who interpret the results. Although such automation can help maintain consistency and increase efficiency in forensic analysis, there are several major limitations that must be guarded against. Automated systems can have bugs that produce incorrect results, which can have serious consequences in a forensic context.<sup id=\"rdp-ebb-cite_ref-MurrayQueens15_9-0\" class=\"reference\"><a href=\"#cite_note-MurrayQueens15-9\">[9]<\/a><\/sup> Additionally, automated AI\/ML systems can introduce bias due to poorly selected training datasets, and can lead to misinterpretations when the results are not fully understood.<sup id=\"rdp-ebb-cite_ref-MargagliottiMachine19_10-0\" class=\"reference\"><a href=\"#cite_note-MargagliottiMachine19-10\">[10]<\/a><\/sup> When automated AI\/ML systems are used to support investigation and forensic analysis, such as comparison of faces in digital video or photographs, algorithmic false positives can lead to incorrect results.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Reliability_and_human_rights_concerns_scenario\">Reliability and human rights concerns scenario<\/span><\/h4>\n<p>In this scenario, an independent study of six facial recognition technology test deployments performed by the London Metropolitan Police Service (MPS) found a high number of false positives. In total, only eight out of 46 automatically generated potential face recognition \u201cmatches\u201d that were considered and evaluated by a human were deemed to be correct. The eight verified correct matches were determined by some form of confirmation such as through a street-based identity check.<sup id=\"rdp-ebb-cite_ref-FusseyIndep19_11-0\" class=\"reference\"><a href=\"#cite_note-FusseyIndep19-11\">[11]<\/a><\/sup> The study paid particular attention to the risks of such technology interfering with fundamental human rights, including privacy violations and discrimination due to algorithmic bias relating to sex, race, or ethnicity.\n<\/p><p>This study highlights the fact that any automated system can have some false positives and false negatives, which demands that human analysts are in the loop to manage the risks of something important being overlooked or misinterpreted. Good practice here can also help deal with the risk of contextual bias. An automated system supporting forensic analysis should offer sequential unmasking capabilities.<sup id=\"rdp-ebb-cite_ref-KopplStrat19_12-0\" class=\"reference\"><a href=\"#cite_note-KopplStrat19-12\">[12]<\/a><\/sup>\n<\/p><p>Ultimately, automated systems supporting forensic analysis should integrate forensic requirements, risk management, and privacy, i.e., forensic-by-design. If a system is forensic-by-design, forensic preparedness principles and practices have been integrated into the system engineering lifecycle, including risk management, incident handling, forensic principles, and legal compliance.<sup id=\"rdp-ebb-cite_ref-RahmanForensic16_13-0\" class=\"reference\"><a href=\"#cite_note-RahmanForensic16-13\">[13]<\/a><\/sup> However, such forensic-by-design is not common practice at the moment, and there are growing concerns about the unreliability and invasiveness of such technologies for criminal investigation purposes. Some cities in California are strictly controlling the acquisition and utilization of \"any software, electronic device, system utilizing an electronic device, or similar device used, designed, or primarily intended to collect, retain, process, or share audio, electronic, visual, location, thermal, biometric, olfactory or similar information specifically associated with, or capable of being associated with, any individual or group.\"<sup id=\"rdp-ebb-cite_ref-SFCountyOrdinance19_14-0\" class=\"reference\"><a href=\"#cite_note-SFCountyOrdinance19-14\">[14]<\/a><\/sup>\n<\/p><p>Forensic laboratories must be prepared to address concerns about reliability, privacy, and discrimination associated with use of technology for processing data, particularly AI\/ML systems for data analysis.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Digital_reinforcement_of_forensic_science_principles\">Digital reinforcement of forensic science principles<\/span><\/h2>\n<p>Core principles and processes in forensic laboratories can be bolstered using technology while increasing the efficiency and quality of services. These core principles include authenticity and integrity, reliability and reproducibility, and quality and efficiency. In addition, intelligent application of digital technology can create new opportunities to deal with crime in digitalized society.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authenticity_and_integrity\">Authenticity and integrity<\/span><\/h3>\n<p>Digital transformation of forensic laboratories can enhance the traceability of traces. Existing computerized systems for processing physical and digital evidence typically provide a digital audit trail and its treatment throughout its lifecycle in the forensic laboratory. Alterations to data files (raw and processed) might be detectable using the digital traces that are created routinely during the processing of exhibits in a forensic laboratory. In some situations, to authenticate data files and associated results, it is necessary to perform contextual analysis, including digital forensic analysis of the original data and metadata.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Contextual_reconstruction_scenario\">Contextual reconstruction scenario<\/span><\/h4>\n<p>In this scenario, questions have arisen about the results of some samples processed by a laboratory. In order to assess the reported results, the raw data was recovered from analysis systems along with processed results. The information from these files was compared with details recorded in the LIMS, and discrepancies were found. However, audit logs in the LIMS were found that corresponded with the original data. Timestamps of the original data files and the LIMS were also examined to determine contemporaneous activities versus later changes.\n<\/p><p>As this scenario highlights, it is advisable not to leave evidence authentication to chance. With properly protected audit trail and data file integrity mechanisms (preferably automated), a digitalized chain of custody can be maintained from evidence intake to evidence return or archiving, providing valuable insights into laboratory operations and helping protect against undetected mistakes and forensic fraud. Some forensic laboratories have adopted bar code scanning and RFID labels to mark and track exhibits throughout their lifecycle. Some forensic laboratories use secure storage to retain raw data files. Some laboratories are developing enhanced methods for maintaining provenance information using blockchain-based systems.<sup id=\"rdp-ebb-cite_ref-BurriChrono20_7-1\" class=\"reference\"><a href=\"#cite_note-BurriChrono20-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Jaquet-ChiffelleTamper20_8-1\" class=\"reference\"><a href=\"#cite_note-Jaquet-ChiffelleTamper20-8\">[8]<\/a><\/sup>\n<\/p><p>With such automated provenance tracking mechanisms in place, every deliverable that a forensic laboratory produces could include the associated electronic chain of custody details as an appendix to demonstrate proper handling. Providing this provenance information in a standardized format such as <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/caseontology.org\/\" target=\"_blank\">CASE<\/a> enables receiving organizations to integrate the information into their information management systems and detect potential inconsistencies more easily, and even automatically.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Reliability_and_reproducibility\">Reliability and reproducibility<\/span><\/h3>\n<p>Forensic science practices demand that the data, methods, tools, and analysis are described in sufficient detail that they can be carried out again with the same results.\n<\/p><p>Digital transformations can support reproducibility of processing by documenting all phases of the evidence lifecycle in the forensic laboratory, as well as reproducibility of analysis by providing others with original data to perform independent analysis. Some software developers also provide a standalone application for viewing the results of forensic processes, enabling others to verify forensic analysis without requiring them to purchase licenses for the specific software and\/or version.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Reproducibility_of_forensic_analysis_scenario\">Reproducibility of forensic analysis scenario<\/span><\/h4>\n<p>In this scenario, multiple parties need to perform an in-depth review of the results produced by a forensic laboratory, as part of an investigation involving various computers and mobile devices. To save cost and time, the forensic laboratory provided the multiple parties with a complete package of data and viewing software necessary to perform in-depth review. Using this approach, all parties could assess the reliability of the evidence and results without having to purchase specialized equipment and software, or to run time consuming data processing operations that had already been performed by the laboratory.\n<\/p><p>This scenarion shows that expanding decentralisation of forensic capabilities is driving the need for data and analysis results to be transferred securely between systems and organizations that is fully traceable, if not completely repeatable. This trend raises the importance of international standards for harmonization of forensic methods and data formats.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Quality_and_efficiency\">Quality and efficiency<\/span><\/h3>\n<p>Technological advances enable forensic laboratories to process evidence more quickly while maintaining quality. Forensic laboratories can manage the systems that they rely on for processing evidence in order to standardize the processing and output. The National Institute of Standards and Technology (NIST) Computer Forensic Tool Testing (CFTT) program publishes open access test reports for software commonly used in digital forensic laboratories, detailing both capabilities and limitations. The European Network of Forensic Science Institutes (ENFSI) and the US DoD Cyber Crime Center (DC3) also perform tool validation and testing, and provide the results to law enforcement agencies. Similar validation and testing should be applied to computers and software used to operate traditional forensic equipment. As with the prior discussed scenario of computer system malfunction, such validation must be alert for seemingly insignificant changes that can create major errors in traditional forensic processes. Furthermore, the validation of AI\/ML based software is an important and complex undertaking that must be performed by an independent entity or consortium. This level of control helps improve the consistency and quality of results from forensic laboratory processes.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Validation_and_change_control_scenario\">Validation and change control scenario<\/span><\/h4>\n<p>In this scenario, image that a forensic laboratory routinely validates and tests equipment, and maintains strict change control procedures for computers used to process evidence. Every aspect of the systems gets covered, including the specific hardware, firmware, and software versions. Validation of the forensic processes gets performed on the specific systems, and any issues are recognized and dealt with before causing further problems.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Forensic-by-design_automation\">Forensic-by-design automation<\/span><\/h3>\n<p>Forensic laboratories are developing new automated systems, including those with AI\/ML capabilities, to analyze data and to gain understanding of crime in ways that were previously not feasible.<sup id=\"rdp-ebb-cite_ref-MargagliottiMachine19_10-1\" class=\"reference\"><a href=\"#cite_note-MargagliottiMachine19-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CaseyTheKodak18_15-0\" class=\"reference\"><a href=\"#cite_note-CaseyTheKodak18-15\">[15]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Forensic_artificial_intelligence_scenario\">Forensic artificial intelligence scenario<\/span><\/h4>\n<p>In this scenario, imagine that electronic portable devices, based on near-infrared spectroscopy and supported by machine learning, have been developed to test drugs rapidly. These devices could be useful for producing a full profile of the substance tested in the field and transfer the data to a computerized repository in the laboratory for further analysis.<sup id=\"rdp-ebb-cite_ref-CoppeyProvid20_16-0\" class=\"reference\"><a href=\"#cite_note-CoppeyProvid20-16\">[16]<\/a><\/sup> Such a repository could be useful for tracking drug trends in a region, and quickly detecting a problem such as an .\n<\/p><p>The value of forensic laboratories can be extended to more proactive intelligence-led approaches based on knowledge extracted from repetitions found in crime.<sup id=\"rdp-ebb-cite_ref-CaseyTheKodak18_15-1\" class=\"reference\"><a href=\"#cite_note-CaseyTheKodak18-15\">[15]<\/a><\/sup> To manage the associated risks, systems supporting such intelligence-based solutions should be forensic-by-design, abiding by forensic principles and human rights, including privacy and nondiscrimination. Forensic-by-design includes traceability, encryption, and impossibility even for system administrators to access private information.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Digital_transformation_risk_management\">Digital transformation risk management<\/span><\/h2>\n<p>This section describes a series of recommendations (Table 1) for managing risks of digital transformation of forensic laboratories. Forensic laboratories need to prepare for matters such as lost evidence, being audited or investigated, and responding to civil lawsuits. In the digital forensic domain, the practice of forensic preparedness has been developed to manage risks associated with computer use and misuse. The practice of forensic preparedness involves specification of a policy that lays down a consistent approach, detailed planning against typical (and actual) audit or investigative scenarios that an organization faces, identification of (internal or external) resources that can be deployed as part of those plans, identification of where and how the associated digital evidence can be gathered that will support investigation, and a process of <a href=\"https:\/\/www.limswiki.org\/index.php\/Continual_improvement_process\" title=\"Continual improvement process\" class=\"wiki-link\" data-key=\"fd7b54be3f6cdd0e8ed84d501486d668\">continuous improvement<\/a> that learns from experience.<sup id=\"rdp-ebb-cite_ref-CESGoodPrac15_17-0\" class=\"reference\"><a href=\"#cite_note-CESGoodPrac15-17\">[17]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 1.<\/b> Road-map for digital transformation risk management\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Risk\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Recommendation\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Associated subsection (below)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not knowing what data exist\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Develop a digital evidence map documenting where needed data, logs, and provenance details are located and how they will be preserved in a forensic manner.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Curated digital information\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not having a plan\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Institute policies and procedures that govern the roles, responsibilities, and expected actions during an inquiry.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Established processes and procedures\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Missing data and metadata\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Implement an automated digital forensic preservation process of primary data sources.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Data integrity\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lack of traceability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Maintain and forensically preserve detailed electronic audit logs of all computer system usage and database transactions.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Strategic audit logging\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lack of data integrity\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tamperproof the digitalized chain of custody ledger to support authentication of data files and audit logs.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Data authentication\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lack of practice\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Use audit logs and digitalized chain of custody records for routine purposes to ensure that the logs are monitored and used regularly.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Regular use or review of data and processes\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Insufficient validation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Validate computer-reliant forensic processes whenever there are changes to hardware or software configurations.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Test and validation\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Black-box automation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Evaluate automated systems at three levels: performance, understandability, and scientific interpretation.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Effective use of automation\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lack of digital forensic support\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Engage digital forensic expertise in preparation for problems, and to assist with dealing with problems that arise in a forensic laboratory.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Role of digital forensic expertise\"\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Lack of forensic preparedness increases the risks of problems going undetected and of ineffective response after a problem is detected. A reactive approach is costly and disruptive, including the need to find and retain external digital forensic expertise. Forensic preparedness enhances business continuity and contingency planning, putting organizations in a better position to detect and investigate problems and manage the associated risks.<sup id=\"rdp-ebb-cite_ref-JohnstonNetwork06_18-0\" class=\"reference\"><a href=\"#cite_note-JohnstonNetwork06-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ElyasDigital15_19-0\" class=\"reference\"><a href=\"#cite_note-ElyasDigital15-19\">[19]<\/a><\/sup> These issues and remedies apply equally to all forensic laboratories, including those within private enterprises.\n<\/p><p>It is important to note that this road map of recommendations does not cover <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cyberattacks<\/a>, which require additional digital forensic preparations such as incident response procedures and expertise.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Curated_digital_information\">Curated digital information<\/span><\/h3>\n<p>A fundamental aspects of being prepared from a digital forensic perspective is knowing where key data sources are located, and ensuring that they contain the minimum data necessary to support business needs and meet legal requirements.\n<\/p><p>The purpose of curating digital information is to reduce the amount of time required to access and examine relevant data, thus reducing the interruption of business continuity and the overall cost to the forensic laboratory. Minimizing interruptions in forensic laboratories is essential for criminal justice, and some countries require annual testing of business continuity plans, with guidance from ISO 22313:2012 <i>Societal security \u2014 Business continuity management systems \u2014 Guidance<\/i>.<sup id=\"rdp-ebb-cite_ref-TullyForensic14_20-0\" class=\"reference\"><a href=\"#cite_note-TullyForensic14-20\">[20]<\/a><\/sup>\n<\/p><p>Curating digital information entails identifying all computer-reliant processes and then augmenting data sources to facilitate authentication of digital evidence (Figure 2). When curating digital information, it is also important to eliminate extraneous data in order to avoid inordinately large repositories that are costly and time-consuming to search or produce.<sup id=\"rdp-ebb-cite_ref-CaseyDigital07_21-0\" class=\"reference\"><a href=\"#cite_note-CaseyDigital07-21\">[21]<\/a><\/sup> \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Casey_ForensicSciInt2020_316.jpg\" class=\"image wiki-link\" data-key=\"e9af6decc22fe69293ba10634a64fbf2\"><img alt=\"Fig2 Casey ForensicSciInt2020 316.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/69\/Fig2_Casey_ForensicSciInt2020_316.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Example digital evidence map for forensic laboratories.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Forensic laboratories cannot rely solely on system administrators to curate digital information sources because these individuals are primarily concerned with system performance and security, and they have limited experience with forensic preparedness. To ensure that forensic preparation will withstand scrutiny, it is advisable to involve experienced digital forensic practitioners in the process.\n<\/p><p>An important aspect of this forensic preservation process is to have a <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a> policy that specifies how long data will be maintained. For instance, a data management strategy could include a control to erase the original files after they are forensically preserved, to avoid having to perform a costly and time-consuming forensic preservation and examination of every computer in the laboratory whenever there is a problem. Data retention decisions must take into account applicable national laws and regulations of the country.\n<\/p><p>One of the most useful tools that investigators can have is a map indicating where evidence is located on a network, i.e., a digital evidence map. Such a map is even more useful when it specifies how long digital evidence remains on the network and references procedures for collecting the evidence.<sup id=\"rdp-ebb-cite_ref-CaseyDigital11_22-0\" class=\"reference\"><a href=\"#cite_note-CaseyDigital11-22\">[22]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Established_processes_and_procedures\">Established processes and procedures<\/span><\/h3>\n<p>Having documented processes to handle common audit and investigative scenarios puts forensic laboratories in a stronger position to respond in an organised, efficient, and effective manner. Effective forensic preparedness requires clear oversight, authorizations, responsibilities, expected actions, desired results, and restrictions. For example, who will gather needed data from various sources in the forensic laboratory (e.g., from the LIMS, computers, backups), how they will perform these tasks, and under whose authority of supervision.\n<\/p><p>The purpose of documented processes and procedures is to ensure a more organized and efficient response and to reduce the risk of mistakes and oversights, thus limiting the associated disruption, liability, and cost for forensic laboratories.\n<\/p><p>These procedures include preservation of digital data with minimal disruption to business continuity and forensic laboratory operations. Proper preservation of digital data supports investigative, forensic, legal, and regulatory requirements, and can help a forensic laboratory defend against any subsequent civil litigation. Preservation is vastly simplified when forensic preparations are already in place.\n<\/p><p>Forensic laboratories should periodically test these processes and procedures to ensure they work as expected.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_integrity\">Data integrity<\/span><\/h3>\n<p>Manufacturers of laboratory equipment focus on the intended use of their systems, rather than potential misuses. Although manufacturers have some ability to check that output data is well formed, they cannot detect all types of alteration or corruption. Therefore, forensic laboratories cannot rely solely on the manufacturers of equipment to detect alteration or corruption of their output data. A straightforward mechanism for assuring the <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">integrity of data<\/a> is to generate cryptographic hash values of files, as demonstrated in the previously discussed data retention and loss scenario. Using a digital forensic preservation method such as AFF4 provides more robust data integrity and evidence management. The integrity information can be stored in a LIMS system along with other pertinent details for an evidence item, or in a tamperproof digitalized chain of custody ledger.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Strategic_audit_logging\">Strategic audit logging<\/span><\/h3>\n<p>Forensic laboratories need to pay careful attention to maintaining chain of custody, including generating and protecting audit logging and mechanisms to ensure the authenticity and integrity of data in a LIMS, as well as generated instrument files (raw and processed).\n<\/p><p>Ready access to audit logs allows faster response to problems and more comprehensive assessment of the scope of a problem, thereby helping reduce the associated interruption and cost. When a problem occurs in a forensic laboratory, it is advisable to make a forensic copy of all audit logs, preserving them as a source of evidence for further analysis. In some situations, it might also be necessary to restore older audit logs from backup to obtain a complete view of relevant activities.\n<\/p><p>Under certain conditions, system administrator level access can be used to bypass many protections and access controls on computer systems. Therefore, additional measures are needed to maintain the integrity of data in forensic laboratories such as maintaining integrity records contemporaneously, as discussed in the next section.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_authentication\">Data authentication<\/span><\/h3>\n<p>To enable authentication of data, forensic results, and audit logs, some forensic laboratories print hard copies of certain data to maintain paper records for future comparison with digital information. Another approach is to archive copies of the original digital data on read-only storage media. Authentication can be further enhanced with a tamper-proof digitalized chain of custody ledger using blockchain infrastructure, as referenced above.\n<\/p><p>Given the importance of temporal information for data authentication, it is important for clocks on all computer systems to be automatically synchronized with a single timesource and timezone.\n<\/p><p>Some laboratories are adopting the CASE specification to represent digital forensic information, including provenance details, in a standardised form, maintaining provenance of exhibits throughout their lifecycle in forensic laboratories.<sup id=\"rdp-ebb-cite_ref-CaseyAdvancing17_23-0\" class=\"reference\"><a href=\"#cite_note-CaseyAdvancing17-23\">[23]<\/a><\/sup> Data marking is an integral part of CASE, supporting data protection for privacy and security purposes.<sup id=\"rdp-ebb-cite_ref-CaseyUsing17_24-0\" class=\"reference\"><a href=\"#cite_note-CaseyUsing17-24\">[24]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Regular_use_or_review_of_data_and_processes\">Regular use or review of data and processes<\/span><\/h3>\n<p>A common mistake that organizations make is to configure data retention and logging and only examine the information after a problem occurs. As a result, problems are registered in preserved data and audit logs but are not observed by anyone in the organization.\n<\/p><p>Routinely relying on the curated data sources, or at least reviewing them to ensure that those responsible are familiar with using and interpreting them, have effective tools for examining them, and promptly notice and resolve failures or errors such as malfunctioning data retention processes, corrupt data, incomplete records, and incorrect time stamps.<sup id=\"rdp-ebb-cite_ref-CaseyIntrusion10_25-0\" class=\"reference\"><a href=\"#cite_note-CaseyIntrusion10-25\">[25]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Test_and_validation\">Test and validation<\/span><\/h3>\n<p>When deploying new automated capabilities to support forensic analysis, it is necessary to test and validate the system to determine its reliability and limitations. Finding and mitigating problems in such systems is non-trivial and requires a systematic approach and specialized expertise.<sup id=\"rdp-ebb-cite_ref-TaylorComm17_26-0\" class=\"reference\"><a href=\"#cite_note-TaylorComm17-26\">[26]<\/a><\/sup> Forensic laboratories have the necessary knowledge and structure to manage quality and risk of complex processes, which can be extended to encompass automated AI\/ML systems.<sup id=\"rdp-ebb-cite_ref-CaseyTheKodak18_15-2\" class=\"reference\"><a href=\"#cite_note-CaseyTheKodak18-15\">[15]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Effective_use_of_automation\">Effective use of automation<\/span><\/h3>\n<p>One way to mitigate the risks associated with automation to support forensic analysis is to realize the value of human expertise. A study of facial comparison found that the optimal results were obtained when forensic expertise was combined with state-of-the-art face recognition technology.<sup id=\"rdp-ebb-cite_ref-PhillipsFace18_27-0\" class=\"reference\"><a href=\"#cite_note-PhillipsFace18-27\">[27]<\/a><\/sup>\n<\/p><p>However, when automated systems are not designed with core forensic principles in mind, lacking transparency, there is a risk of the black-box effect.<sup id=\"rdp-ebb-cite_ref-MargagliottiMachine19_10-2\" class=\"reference\"><a href=\"#cite_note-MargagliottiMachine19-10\">[10]<\/a><\/sup> To manage this risk, forensic laboratories need to ensure that all automated processes they rely on produce forensic results, including AI\/ML based systems, can be explained by the laboratory specialists. Limitations of automated systems, including possible false-positives and false-negatives, should be documented to manage risks of errors, omissions, and misuse.\n<\/p><p>To address the risk of incorrect decisions based on automated systems, it is necessary for such systems to function well at three levels: performance, understandability, and scientific interpretation.<sup id=\"rdp-ebb-cite_ref-Boll.C3.A9TheRole20_28-0\" class=\"reference\"><a href=\"#cite_note-Boll.C3.A9TheRole20-28\">[28]<\/a><\/sup> It is important to address these issues as part of forensic preparedness, before a problem occurs. Such preparations will put forensic laboratories in a stronger position to defend decisions based on an automatic system and explain the underlying logic.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Role_of_digital_forensic_expertise\">Role of digital forensic expertise<\/span><\/h2>\n<p>Success in managing risks associated with digital transformations in forensic laboratories depends on the qualifications and experience of the personnel performing the digital forensic processes.\n<\/p><p>Although there are ongoing efforts to harmonize digital evidence and forensic science<sup id=\"rdp-ebb-cite_ref-PollittAFrame19_1-1\" class=\"reference\"><a href=\"#cite_note-PollittAFrame19-1\">[1]<\/a><\/sup>, some forensic laboratories are currently unable to integrate digital forensic science, due to division of responsibilities or unavailability of resources and expertise. To help manage the risks associated with digital transformations, forensic laboratories can engage external digital forensic expertise to help develop forensic digital preparedness and respond to problems. It is most effective to arrange this prior to a problem occurring.\n<\/p><p>When a problem occurs, it can be larger than the organization initially realizes. Forensic preparedness makes it easier to determine the full scope of the problem and manage the potential damage more effectively. However, even with perfect preparation, there are usually unexpected challenges in any digital investigation. When the people dealing with a problem are not properly trained in digital forensic science, there is an increased risk of misunderstanding and misinterpretation. Even when they perform their work impeccably, unclear explanation can cause decision makers to miscomprehend digital forensic conclusions.<sup id=\"rdp-ebb-cite_ref-CaseyTrust19_29-0\" class=\"reference\"><a href=\"#cite_note-CaseyTrust19-29\">[29]<\/a><\/sup> The knowledge and experience required to effectively analyze and evaluate digital forensic evidence should not be underestimated.<sup id=\"rdp-ebb-cite_ref-CaseyForensic20_30-0\" class=\"reference\"><a href=\"#cite_note-CaseyForensic20-30\">[30]<\/a><\/sup>\n<\/p><p>Furthermore, digital forensic analysis can reveal weaknesses in forensic laboratory operations, and digital forensic expertise can help continuously improve forensic processes.<sup id=\"rdp-ebb-cite_ref-CaseyForensic20_30-1\" class=\"reference\"><a href=\"#cite_note-CaseyForensic20-30\">[30]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Quality_assurance_considerations\">Quality assurance considerations<\/span><\/h2>\n<p>The forensic preparedness measures discussed above not only help laboratories respond more effectively and efficiently to problems, they also help improve quality assurance and auditability. A digital evidence map, digitally strengthened chain of custody ledger, and strategic audit logging provide more detail and transparency into the handling of exhibits in forensic laboratories, and enables more effective problem detection and response.\n<\/p><p>It is also inadvisable to use the forensic laboratory LIMS to manage evidence of an investigation into its own problems. Therefore, forensic laboratories need to have a separate system for maintaining documentation, incident details, and evidence provenance, and other information that needs to be recorded during investigation of an incident.\n<\/p><p>This raises the question of whether there is a need to incorporate the digital transformation risk mitigation measures discussed in this paper into standards for accreditation such as ISO\/IEC 17025:2017\n<i>General requirements for the competence of testing and calibration laboratories<\/i>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Current_requirements\">Current requirements<\/span><\/h3>\n<p>Validation within forensic laboratories must include the underlying technology, including their hardware and software configuration, access control (security), and auditability. Each time there is a significant change in hardware or software, routine tests and validation should be performed against known datasets to determine whether expected results are produced.\n<\/p><p>Nevertheless, according to the authors\u2019 experiences with laboratories both in Europe and the U.S., even ISO 17025 accredited laboratories do not always perform a validation process for each major release of every software application used for forensic analysis. This lack of rigour can be explained by the difficulty to trigger significant changes but is most of the time due to time-saving reasons.\n<\/p><p>Moreover, if the ISO 17025 standard requires information management systems to be validated, the current version still states that \u201ccommercial off-the-shelf software in general use within its designed application range can be considered to be sufficiently validated.\u201d<sup id=\"rdp-ebb-cite_ref-ISOIEC17025_31-0\" class=\"reference\"><a href=\"#cite_note-ISOIEC17025-31\">[31]<\/a><\/sup> This large exception raises questions about the existence and quality of validation processes performed by system providers. Some laboratories might also consider only validating internal systems.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Possible_improvements\">Possible improvements<\/span><\/h3>\n<p>To overcome the limits related to validation cost of commercial software, independent bodies could share their validation results, following the digital forensic tool testing approach taken by NIST and DC3.\n<\/p><p>The need for such validation and testing does not only apply to systems used to process exhibits. More attention could be given to validation of LIMS systems, including their reliability, security, and auditability. Forensic laboratories should also ensure that access controls are in place on LIMS and secure storage systems to prevent unauthorized alteration or deletion. This issue could be address with clearer wording in quality standards such as ISO 17025, or in the the international guidance document ILAC G19.\n<\/p><p>The risks of oversights and misinterpretations are growing as more complex automated systems being used for forensic purposes lack forensics-by-design. To mitigate the associated risks, forensic laboratories need to implement additional measures that govern proper operation and use of automated systems.\n<\/p><p>Quality standards such as ISO 17025 could be further refined by adding specific requirements for forensic digital preparedness described in this paper.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Forensic laboratories that fail to seriously confront digital transformation risk management will suffer significant disruption and expense when problems arise. To mitigate these risks, forensic laboratories must strengthen their forensic digital preparedness and ensure that technology abides by core principles and processes, i.e., authenticity and integrity, reliability and reproducibility, and quality and efficiency. Laboratories should also consider applying forensic digital preparedness to their email and other administrative systems. As much as feasible, the recommendations in this paper are \u201cenable and forget\u201d (until a problem occurs). An initial investment in forensic preparation can return repeated benefits by reinforcing forensic principles, as noted in the section \"Digital reinforcement of forensic science principles,\" and by reducing expenses and business disruption each time a problem arises.\n<\/p><p>In brief, laboratories should take forensic digital preparedness steps before a problem arises, and consider involving digital forensic specialists with experience in forensic laboratory operations. When a problem occurs, follow documented processes and procedures for responding to such incidents. If a plan does not exist, create one and implement it methodically. Know where sources of relevant digital data are, and take steps to preserve them properly. Forensic preservation of data includes original files and backups, as well as audit logs. Perform a thorough scope assessment to determine the full extent of the problem. Document all actions taken in response to the problem.\n<\/p><p>It would be generally beneficial to require LIMS and other digitalized processes in forensic laboratories to be forensic-by-design. The recommendations in this paper provide a foundation for forensic laboratories to develop requirements that system providers should fulfill.\n<\/p><p>With proper forethought and preparation, forensic laboratories can employ technology and advanced data analytics to enhance existing services and create new services, while respecting fundamental human rights.\n<\/p><p>Applying their existing knowledge and structures, forensic laboratories are in a strong position to effectively manage quality and risk of digital transformations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>Our deepest gratitude to Christophe Champod, Timothy Boll\u00e9, Francesco Servida, and Hannes Spichiger for their collaboration and insights. We are also thankful for the peer reviewers' recommendations, which improved this work.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Conception and design of study: Eoghan Casey, Thomas Souvignet. Drafting the manuscript: Eoghan Casey, Thomas Souvignet. Approval of the version of the manuscript to be published: Eoghan Casey, Thomas Souvignet.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h3>\n<p>None stated.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-PollittAFrame19-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PollittAFrame19_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-PollittAFrame19_1-1\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pollitt, M.; Casey, E.; Jaquet-Chiffelle, D.-O. et al. (February 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nist.gov\/system\/files\/documents\/2018\/01\/10\/osac_ts_0002.pdf\" target=\"_blank\">\"A Framework for Harmonizing Forensic Science Practices and Digital\/Multimedia Evidence\"<\/a> (PDF). OSAC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nist.gov\/system\/files\/documents\/2018\/01\/10\/osac_ts_0002.pdf\" target=\"_blank\">https:\/\/www.nist.gov\/system\/files\/documents\/2018\/01\/10\/osac_ts_0002.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+Framework+for+Harmonizing+Forensic+Science+Practices+and+Digital%2FMultimedia+Evidence&rft.atitle=&rft.aulast=Pollitt%2C+M.%3B+Casey%2C+E.%3B+Jaquet-Chiffelle%2C+D.-O.+et+al.&rft.au=Pollitt%2C+M.%3B+Casey%2C+E.%3B+Jaquet-Chiffelle%2C+D.-O.+et+al.&rft.date=February+2019&rft.pub=OSAC&rft_id=https%3A%2F%2Fwww.nist.gov%2Fsystem%2Ffiles%2Fdocuments%2F2018%2F01%2F10%2Fosac_ts_0002.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TullyQuality20-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TullyQuality20_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tully, G.; Cohen, N.; Compton, D. et al. (2020). \"Quality standards for digital forensics: Learning from experience in England & Wales\". <i>Forensic Science International: Digital Investigation<\/i> <b>32<\/b>: 200905. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.fsidi.2020.200905\" target=\"_blank\">10.1016\/j.fsidi.2020.200905<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quality+standards+for+digital+forensics%3A+Learning+from+experience+in+England+%26+Wales&rft.jtitle=Forensic+Science+International%3A+Digital+Investigation&rft.aulast=Tully%2C+G.%3B+Cohen%2C+N.%3B+Compton%2C+D.+et+al.&rft.au=Tully%2C+G.%3B+Cohen%2C+N.%3B+Compton%2C+D.+et+al.&rft.date=2020&rft.volume=32&rft.pages=200905&rft_id=info:doi\/10.1016%2Fj.fsidi.2020.200905&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ReustIdent08-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ReustIdent08_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Reust, J.; Sommers, R.; Friedberg, S. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20160429004454\/https:\/\/www.aafs.org\/wp-content\/uploads\/ProceedingsWashingtonDC2008.pdf\" target=\"_blank\">\"Identification and Reconstruction of Deleted, Fragmented DNA Digital Files\"<\/a>. <i>Proceedings of the American Academy of Forensic Sciences<\/i> <b>14<\/b>: 187\u201388. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.aafs.org\/wp-content\/uploads\/ProceedingsWashingtonDC2008.pdf\" target=\"_blank\">the original<\/a> on 29 April 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20160429004454\/https:\/\/www.aafs.org\/wp-content\/uploads\/ProceedingsWashingtonDC2008.pdf\" target=\"_blank\">https:\/\/web.archive.org\/web\/20160429004454\/https:\/\/www.aafs.org\/wp-content\/uploads\/ProceedingsWashingtonDC2008.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identification+and+Reconstruction+of+Deleted%2C+Fragmented+DNA+Digital+Files&rft.jtitle=Proceedings+of+the+American+Academy+of+Forensic+Sciences&rft.aulast=Reust%2C+J.%3B+Sommers%2C+R.%3B+Friedberg%2C+S.+et+al.&rft.au=Reust%2C+J.%3B+Sommers%2C+R.%3B+Friedberg%2C+S.+et+al.&rft.date=2008&rft.volume=14&rft.pages=187%E2%80%9388&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20160429004454%2Fhttps%3A%2F%2Fwww.aafs.org%2Fwp-content%2Fuploads%2FProceedingsWashingtonDC2008.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CohenExtend09-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CohenExtend09_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cohen, M.; Garfinkel, S.; Schatz, B. (2009). \"Extending the advanced forensic format to accommodate multiple data sources, logical evidence, arbitrary information and forensic workflow\". <i>Digital Investigation<\/i> <b>6<\/b> (Supplement 1): S57\u2013S68. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.diin.2009.06.010\" target=\"_blank\">10.1016\/j.diin.2009.06.010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Extending+the+advanced+forensic+format+to+accommodate+multiple+data+sources%2C+logical+evidence%2C+arbitrary+information+and+forensic+workflow&rft.jtitle=Digital+Investigation&rft.aulast=Cohen%2C+M.%3B+Garfinkel%2C+S.%3B+Schatz%2C+B.&rft.au=Cohen%2C+M.%3B+Garfinkel%2C+S.%3B+Schatz%2C+B.&rft.date=2009&rft.volume=6&rft.issue=Supplement+1&rft.pages=S57%E2%80%93S68&rft_id=info:doi\/10.1016%2Fj.diin.2009.06.010&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchatzWire15-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchatzWire15_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schatz, B.L. (2015). \"Wirespeed: Extending the AFF4 forensic container format for scalable acquisition and live analysis\". <i>Digital Investigation<\/i> <b>14<\/b> (Supplement 1): S45\u201354. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.diin.2015.05.016\" target=\"_blank\">10.1016\/j.diin.2015.05.016<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Wirespeed%3A+Extending+the+AFF4+forensic+container+format+for+scalable+acquisition+and+live+analysis&rft.jtitle=Digital+Investigation&rft.aulast=Schatz%2C+B.L.&rft.au=Schatz%2C+B.L.&rft.date=2015&rft.volume=14&rft.issue=Supplement+1&rft.pages=S45%E2%80%9354&rft_id=info:doi\/10.1016%2Fj.diin.2015.05.016&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BidgoodChemist17-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BidgoodChemist17_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bidgood, J. (18 April 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20170419100845\/https:\/\/www.nytimes.com\/2017\/04\/18\/us\/chemist-drug-cases-dismissal.html\" target=\"_blank\">\"Chemist\u2019s Misconduct Is Likely to Void 20,000 Massachusetts Drug Cases\"<\/a>. <i>The New York Times<\/i>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nytimes.com\/2017\/04\/18\/us\/chemist-drug-cases-dismissal.html\" target=\"_blank\">the original<\/a> on 19 April 2017<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20170419100845\/https:\/\/www.nytimes.com\/2017\/04\/18\/us\/chemist-drug-cases-dismissal.html\" target=\"_blank\">https:\/\/web.archive.org\/web\/20170419100845\/https:\/\/www.nytimes.com\/2017\/04\/18\/us\/chemist-drug-cases-dismissal.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chemist%E2%80%99s+Misconduct+Is+Likely+to+Void+20%2C000+Massachusetts+Drug+Cases&rft.atitle=The+New+York+Times&rft.aulast=Bidgood%2C+J.&rft.au=Bidgood%2C+J.&rft.date=18+April+2017&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20170419100845%2Fhttps%3A%2F%2Fwww.nytimes.com%2F2017%2F04%2F18%2Fus%2Fchemist-drug-cases-dismissal.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BurriChrono20-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BurriChrono20_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-BurriChrono20_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Burri, X.; Casey, E.; Boll\u00e9, T. et al. (2020). \"Chronological independently verifiable electronic chain of custody ledger using blockchain technology\". <i>Forensic Science International: Digital Investigation<\/i> <b>33<\/b>: 300976. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.fsidi.2020.300976\" target=\"_blank\">10.1016\/j.fsidi.2020.300976<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Chronological+independently+verifiable+electronic+chain+of+custody+ledger+using+blockchain+technology&rft.jtitle=Forensic+Science+International%3A+Digital+Investigation&rft.aulast=Burri%2C+X.%3B+Casey%2C+E.%3B+Boll%C3%A9%2C+T.+et+al.&rft.au=Burri%2C+X.%3B+Casey%2C+E.%3B+Boll%C3%A9%2C+T.+et+al.&rft.date=2020&rft.volume=33&rft.pages=300976&rft_id=info:doi\/10.1016%2Fj.fsidi.2020.300976&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Jaquet-ChiffelleTamper20-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Jaquet-ChiffelleTamper20_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-Jaquet-ChiffelleTamper20_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jaquet-Chiffelle, D.-O.; Casey, E.; Bourquenoud, J. (2020). \"Tamperproof timestamped provenance ledger using blockchain technology\". <i>Forensic Science International: Digital Investigation<\/i> <b>33<\/b>: 300977. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.fsidi.2020.300977\" target=\"_blank\">10.1016\/j.fsidi.2020.300977<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Tamperproof+timestamped+provenance+ledger+using+blockchain+technology&rft.jtitle=Forensic+Science+International%3A+Digital+Investigation&rft.aulast=Jaquet-Chiffelle%2C+D.-O.%3B+Casey%2C+E.%3B+Bourquenoud%2C+J.&rft.au=Jaquet-Chiffelle%2C+D.-O.%3B+Casey%2C+E.%3B+Bourquenoud%2C+J.&rft.date=2020&rft.volume=33&rft.pages=300977&rft_id=info:doi\/10.1016%2Fj.fsidi.2020.300977&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MurrayQueens15-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MurrayQueens15_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Murray, D. (20 March 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.couriermail.com.au\/news\/queensland\/queensland-authorities-confirm-miscode-affects-dna-evidence-in-criminal-cases\/news-story\/833c580d3f1c59039efd1a2ef55af92b\" target=\"_blank\">\"Queensland authorities confirm \u2018miscode\u2019 affects DNA evidence in criminal cases\"<\/a>. <i>The Courier Mail<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.couriermail.com.au\/news\/queensland\/queensland-authorities-confirm-miscode-affects-dna-evidence-in-criminal-cases\/news-story\/833c580d3f1c59039efd1a2ef55af92b\" target=\"_blank\">https:\/\/www.couriermail.com.au\/news\/queensland\/queensland-authorities-confirm-miscode-affects-dna-evidence-in-criminal-cases\/news-story\/833c580d3f1c59039efd1a2ef55af92b<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Queensland+authorities+confirm+%E2%80%98miscode%E2%80%99+affects+DNA+evidence+in+criminal+cases&rft.atitle=The+Courier+Mail&rft.aulast=Murray%2C+D.&rft.au=Murray%2C+D.&rft.date=20+March+2015&rft_id=https%3A%2F%2Fwww.couriermail.com.au%2Fnews%2Fqueensland%2Fqueensland-authorities-confirm-miscode-affects-dna-evidence-in-criminal-cases%2Fnews-story%2F833c580d3f1c59039efd1a2ef55af92b&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MargagliottiMachine19-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MargagliottiMachine19_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-MargagliottiMachine19_10-1\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-MargagliottiMachine19_10-2\">10.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Margagliotti, G.; Boll\u00e9, T. (2019). \"Machine learning & forensic science\". <i>Forensic Science International<\/i> <b>298<\/b>: 138\u201339. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.forsciint.2019.02.045\" target=\"_blank\">10.1016\/j.forsciint.2019.02.045<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Machine+learning+%26+forensic+science&rft.jtitle=Forensic+Science+International&rft.aulast=Margagliotti%2C+G.%3B+Boll%C3%A9%2C+T.&rft.au=Margagliotti%2C+G.%3B+Boll%C3%A9%2C+T.&rft.date=2019&rft.volume=298&rft.pages=138%E2%80%9339&rft_id=info:doi\/10.1016%2Fj.forsciint.2019.02.045&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FusseyIndep19-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FusseyIndep19_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Fussey, P.; Murray, D. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/repository.essex.ac.uk\/24946\/\" target=\"_blank\">\"Independent Report on the London Metropolitan Police Service\u2019s Trial of Live Facial Recognition Technology\"<\/a>. University of Essex Human Rights Centre<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/repository.essex.ac.uk\/24946\/\" target=\"_blank\">http:\/\/repository.essex.ac.uk\/24946\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Independent+Report+on+the+London+Metropolitan+Police+Service%E2%80%99s+Trial+of+Live+Facial+Recognition+Technology&rft.atitle=&rft.aulast=Fussey%2C+P.%3B+Murray%2C+D.&rft.au=Fussey%2C+P.%3B+Murray%2C+D.&rft.date=2019&rft.pub=University+of+Essex+Human+Rights+Centre&rft_id=http%3A%2F%2Frepository.essex.ac.uk%2F24946%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KopplStrat19-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KopplStrat19_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Koppl, R. (2019). \"Strategic choice in linear sequential unmasking\". <i>Science & Justice<\/i> <b>59<\/b> (2): 166-171. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.scijus.2018.10.010\" target=\"_blank\">10.1016\/j.scijus.2018.10.010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Strategic+choice+in+linear+sequential+unmasking&rft.jtitle=Science+%26+Justice&rft.aulast=Koppl%2C+R.&rft.au=Koppl%2C+R.&rft.date=2019&rft.volume=59&rft.issue=2&rft.pages=166-171&rft_id=info:doi\/10.1016%2Fj.scijus.2018.10.010&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RahmanForensic16-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RahmanForensic16_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rahman, N.H.A.; Glisson, W.B.; Yang, Y. et al. (2016). \"Forensic-by-Design Framework for Cyber-Physical Cloud Systems\". <i>IEEE Cloud Computing<\/i> <b>3<\/b> (1): 50\u201359. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FMCC.2016.5\" target=\"_blank\">10.1109\/MCC.2016.5<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Forensic-by-Design+Framework+for+Cyber-Physical+Cloud+Systems&rft.jtitle=IEEE+Cloud+Computing&rft.aulast=Rahman%2C+N.H.A.%3B+Glisson%2C+W.B.%3B+Yang%2C+Y.+et+al.&rft.au=Rahman%2C+N.H.A.%3B+Glisson%2C+W.B.%3B+Yang%2C+Y.+et+al.&rft.date=2016&rft.volume=3&rft.issue=1&rft.pages=50%E2%80%9359&rft_id=info:doi\/10.1109%2FMCC.2016.5&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SFCountyOrdinance19-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SFCountyOrdinance19_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/sfbos.org\/sites\/default\/files\/o0107-19.pdf\" target=\"_blank\">\"Administrative Code - Acquisition of Surveillance Technology, Ordinance No. 107-19\"<\/a> (PDF). County of San Francisco. 21 May 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/sfbos.org\/sites\/default\/files\/o0107-19.pdf\" target=\"_blank\">https:\/\/sfbos.org\/sites\/default\/files\/o0107-19.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Administrative+Code+-+Acquisition+of+Surveillance+Technology%2C+Ordinance+No.+107-19&rft.atitle=&rft.date=21+May+2019&rft.pub=County+of+San+Francisco&rft_id=https%3A%2F%2Fsfbos.org%2Fsites%2Fdefault%2Ffiles%2Fo0107-19.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyTheKodak18-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CaseyTheKodak18_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-CaseyTheKodak18_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-CaseyTheKodak18_15-2\">15.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Casey, E.; Ribaux, O.; Roux, C. (2019). \"The Kodak Syndrome: Risks and Opportunities Created by Decentralization of Forensic Capabilities\". <i>Journal of Forensic Sciences<\/i> <b>64<\/b> (1): 127\u201336. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2F1556-4029.13849\" target=\"_blank\">10.1111\/1556-4029.13849<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Kodak+Syndrome%3A+Risks+and+Opportunities+Created+by+Decentralization+of+Forensic+Capabilities&rft.jtitle=Journal+of+Forensic+Sciences&rft.aulast=Casey%2C+E.%3B+Ribaux%2C+O.%3B+Roux%2C+C.&rft.au=Casey%2C+E.%3B+Ribaux%2C+O.%3B+Roux%2C+C.&rft.date=2019&rft.volume=64&rft.issue=1&rft.pages=127%E2%80%9336&rft_id=info:doi\/10.1111%2F1556-4029.13849&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoppeyProvid20-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CoppeyProvid20_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coppey, F.; B\u00e9cue, A.; Sacr\u00e9, P.-Y. et al. (2020). \"Providing illicit drugs results in five seconds using ultra-portable NIR technology: An opportunity for forensic laboratories to cope with the trend toward the decentralization of forensic capabilities\". <i>Forensic Science International<\/i> <b>317<\/b>: 110498. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.forsciint.2020.110498\" target=\"_blank\">10.1016\/j.forsciint.2020.110498<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Providing+illicit+drugs+results+in+five+seconds+using+ultra-portable+NIR+technology%3A+An+opportunity+for+forensic+laboratories+to+cope+with+the+trend+toward+the+decentralization+of+forensic+capabilities&rft.jtitle=Forensic+Science+International&rft.aulast=Coppey%2C+F.%3B+B%C3%A9cue%2C+A.%3B+Sacr%C3%A9%2C+P.-Y.+et+al.&rft.au=Coppey%2C+F.%3B+B%C3%A9cue%2C+A.%3B+Sacr%C3%A9%2C+P.-Y.+et+al.&rft.date=2020&rft.volume=317&rft.pages=110498&rft_id=info:doi\/10.1016%2Fj.forsciint.2020.110498&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CESGoodPrac15-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CESGoodPrac15_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">CESG (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/vdocuments.mx\/download\/good-practice-guide-forensic-readiness-ncsc-site-18aa-forensic-readiness\" target=\"_blank\">\"Forensic Readiness\"<\/a>. <i>Good Practice Guide<\/i> (1.2)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/vdocuments.mx\/download\/good-practice-guide-forensic-readiness-ncsc-site-18aa-forensic-readiness\" target=\"_blank\">https:\/\/vdocuments.mx\/download\/good-practice-guide-forensic-readiness-ncsc-site-18aa-forensic-readiness<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Forensic+Readiness&rft.jtitle=Good+Practice+Guide&rft.aulast=CESG&rft.au=CESG&rft.date=2015&rft.issue=1.2&rft_id=https%3A%2F%2Fvdocuments.mx%2Fdownload%2Fgood-practice-guide-forensic-readiness-ncsc-site-18aa-forensic-readiness&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JohnstonNetwork06-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JohnstonNetwork06_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Johnston, A.; Reust, J. (2006). \"Network intrusion investigation \u2013 Preparation and challenges\". <i>Digital Investigation<\/i> <b>3<\/b> (3): 118\u201326. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.diin.2006.08.001\" target=\"_blank\">10.1016\/j.diin.2006.08.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Network+intrusion+investigation+%E2%80%93+Preparation+and+challenges&rft.jtitle=Digital+Investigation&rft.aulast=Johnston%2C+A.%3B+Reust%2C+J.&rft.au=Johnston%2C+A.%3B+Reust%2C+J.&rft.date=2006&rft.volume=3&rft.issue=3&rft.pages=118%E2%80%9326&rft_id=info:doi\/10.1016%2Fj.diin.2006.08.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElyasDigital15-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ElyasDigital15_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Elyas, M.; Ahmad, A.; Maynard, S.B. et al. (2015). \"Digital forensic readiness: Expert perspectives on a theoretical framework\". <i>Computers & Security<\/i> <b>52<\/b>: 70\u201389. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cose.2015.04.003\" target=\"_blank\">10.1016\/j.cose.2015.04.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Digital+forensic+readiness%3A+Expert+perspectives+on+a+theoretical+framework&rft.jtitle=Computers+%26+Security&rft.aulast=Elyas%2C+M.%3B+Ahmad%2C+A.%3B+Maynard%2C+S.B.+et+al.&rft.au=Elyas%2C+M.%3B+Ahmad%2C+A.%3B+Maynard%2C+S.B.+et+al.&rft.date=2015&rft.volume=52&rft.pages=70%E2%80%9389&rft_id=info:doi\/10.1016%2Fj.cose.2015.04.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TullyForensic14-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TullyForensic14_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Tully, G. (05 September 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.gov.uk\/government\/collections\/forensic-science-providers-codes-of-practice-and-conduct\" target=\"_blank\">\"Forensic science providers: codes of practice and conduct\"<\/a>. <i>Gov.UK<\/i>. UK Government<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.gov.uk\/government\/collections\/forensic-science-providers-codes-of-practice-and-conduct\" target=\"_blank\">https:\/\/www.gov.uk\/government\/collections\/forensic-science-providers-codes-of-practice-and-conduct<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Forensic+science+providers%3A+codes+of+practice+and+conduct&rft.atitle=Gov.UK&rft.aulast=Tully%2C+G.&rft.au=Tully%2C+G.&rft.date=05+September+2014&rft.pub=UK+Government&rft_id=https%3A%2F%2Fwww.gov.uk%2Fgovernment%2Fcollections%2Fforensic-science-providers-codes-of-practice-and-conduct&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyDigital07-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CaseyDigital07_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Casey, E. (2007). \"Digital evidence maps \u2013 A sign of the times\". <i>Digital Investigation<\/i> <b>4<\/b> (1): 1\u20132. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.diin.2007.01.004\" target=\"_blank\">10.1016\/j.diin.2007.01.004<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Digital+evidence+maps+%E2%80%93+A+sign+of+the+times&rft.jtitle=Digital+Investigation&rft.aulast=Casey%2C+E.&rft.au=Casey%2C+E.&rft.date=2007&rft.volume=4&rft.issue=1&rft.pages=1%E2%80%932&rft_id=info:doi\/10.1016%2Fj.diin.2007.01.004&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyDigital11-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CaseyDigital11_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Casey, E. (2011). <i>Digital Evidence and Computer Crime: Forensic Science, Computers, and the Internet<\/i>. Academic Press, Inc. pp. 840. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780123742681.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Digital+Evidence+and+Computer+Crime%3A+Forensic+Science%2C+Computers%2C+and+the+Internet&rft.aulast=Casey%2C+E.&rft.au=Casey%2C+E.&rft.date=2011&rft.pages=pp.%26nbsp%3B840&rft.pub=Academic+Press%2C+Inc&rft.isbn=9780123742681&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyAdvancing17-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CaseyAdvancing17_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Casey, E.; Barnum, S.; Griffith, R. et al. (2017). \"Advancing coordinated cyber-investigations and tool interoperability using a community developed specification language\". <i>Digital Investigation<\/i> <b>22<\/b>: 14\u201345. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.diin.2017.08.002\" target=\"_blank\">10.1016\/j.diin.2017.08.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Advancing+coordinated+cyber-investigations+and+tool+interoperability+using+a+community+developed+specification+language&rft.jtitle=Digital+Investigation&rft.aulast=Casey%2C+E.%3B+Barnum%2C+S.%3B+Griffith%2C+R.+et+al.&rft.au=Casey%2C+E.%3B+Barnum%2C+S.%3B+Griffith%2C+R.+et+al.&rft.date=2017&rft.volume=22&rft.pages=14%E2%80%9345&rft_id=info:doi\/10.1016%2Fj.diin.2017.08.002&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyUsing17-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CaseyUsing17_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Casey, E.; Biasiotti, M.A.; Turchi, F. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/serval.unil.ch\/resource\/serval:BIB_EFAFD05944CB.P001\/REF.pdf\" target=\"_blank\">\"Using Standardization and Ontology to Enhance Data Protection and Intelligent Analysis of Electronic Evidence\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/serval.unil.ch\/resource\/serval:BIB_EFAFD05944CB.P001\/REF.pdf\" target=\"_blank\">https:\/\/serval.unil.ch\/resource\/serval:BIB_EFAFD05944CB.P001\/REF.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Using+Standardization+and+Ontology+to+Enhance+Data+Protection+and+Intelligent+Analysis+of+Electronic+Evidence&rft.atitle=&rft.aulast=Casey%2C+E.%3B+Biasiotti%2C+M.A.%3B+Turchi%2C+F.&rft.au=Casey%2C+E.%3B+Biasiotti%2C+M.A.%3B+Turchi%2C+F.&rft.date=2017&rft_id=https%3A%2F%2Fserval.unil.ch%2Fresource%2Fserval%3ABIB_EFAFD05944CB.P001%2FREF.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyIntrusion10-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CaseyIntrusion10_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Casey, E.; Daywalt, C.; Johnston, A. (2010). \"Chapter 4: Intrusion Investigation\". In Casey, E.. <i>Handbook of Digital Forensics and Investigation<\/i>. Academic Press, Inc. pp. 135\u2013206. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FB978-0-12-374267-4.00004-5\" target=\"_blank\">10.1016\/B978-0-12-374267-4.00004-5<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780123742674.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+4%3A+Intrusion+Investigation&rft.atitle=Handbook+of+Digital+Forensics+and+Investigation&rft.aulast=Casey%2C+E.%3B+Daywalt%2C+C.%3B+Johnston%2C+A.&rft.au=Casey%2C+E.%3B+Daywalt%2C+C.%3B+Johnston%2C+A.&rft.date=2010&rft.pages=pp.%26nbsp%3B135%E2%80%93206&rft.pub=Academic+Press%2C+Inc&rft_id=info:doi\/10.1016%2FB978-0-12-374267-4.00004-5&rft.isbn=9780123742674&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorComm17-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorComm17_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, D.A.; Bright, J.-A.; Buckleton, J. (2017). \"Commentary: A \u201cSource\u201d of Error: Computer Code, Criminal Defendants, and the Constitution\". <i>Frontiers in Genetics<\/i> <b>8<\/b>: 33. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffgene.2017.00033\" target=\"_blank\">10.3389\/fgene.2017.00033<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Commentary%3A+A+%E2%80%9CSource%E2%80%9D+of+Error%3A+Computer+Code%2C+Criminal+Defendants%2C+and+the+Constitution&rft.jtitle=Frontiers+in+Genetics&rft.aulast=Taylor%2C+D.A.%3B+Bright%2C+J.-A.%3B+Buckleton%2C+J.&rft.au=Taylor%2C+D.A.%3B+Bright%2C+J.-A.%3B+Buckleton%2C+J.&rft.date=2017&rft.volume=8&rft.pages=33&rft_id=info:doi\/10.3389%2Ffgene.2017.00033&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PhillipsFace18-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PhillipsFace18_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Phillips, P.J.; Yates, A.N.; Hu, Y. et al. (2018). \"Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms\". <i>PNAS<\/i> <b>115<\/b> (24): 6171\u201376. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1073%2Fpnas.1721355115\" target=\"_blank\">10.1073\/pnas.1721355115<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Face+recognition+accuracy+of+forensic+examiners%2C+superrecognizers%2C+and+face+recognition+algorithms&rft.jtitle=PNAS&rft.aulast=Phillips%2C+P.J.%3B+Yates%2C+A.N.%3B+Hu%2C+Y.+et+al.&rft.au=Phillips%2C+P.J.%3B+Yates%2C+A.N.%3B+Hu%2C+Y.+et+al.&rft.date=2018&rft.volume=115&rft.issue=24&rft.pages=6171%E2%80%9376&rft_id=info:doi\/10.1073%2Fpnas.1721355115&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Boll.C3.A9TheRole20-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Boll.C3.A9TheRole20_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Boll\u00e9, T.; Casey, E.; Jacquet, M. et al. (2020). \"The role of evaluations in reaching decisions using automated systems supporting forensic analysis\". <i>Forensic Science International: Digital Investigation<\/i> <b>34<\/b>: 301016. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.fsidi.2020.301016\" target=\"_blank\">10.1016\/j.fsidi.2020.301016<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+role+of+evaluations+in+reaching+decisions+using+automated+systems+supporting+forensic+analysis&rft.jtitle=Forensic+Science+International%3A+Digital+Investigation&rft.aulast=Boll%C3%A9%2C+T.%3B+Casey%2C+E.%3B+Jacquet%2C+M.+et+al.&rft.au=Boll%C3%A9%2C+T.%3B+Casey%2C+E.%3B+Jacquet%2C+M.+et+al.&rft.date=2020&rft.volume=34&rft.pages=301016&rft_id=info:doi\/10.1016%2Fj.fsidi.2020.301016&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyTrust19-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CaseyTrust19_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Casey, E. (2019). \"Trust in digital evidence\". <i>Digital Investigation<\/i> <b>31<\/b>: 200898. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.fsidi.2019.200898\" target=\"_blank\">10.1016\/j.fsidi.2019.200898<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Trust+in+digital+evidence&rft.jtitle=Digital+Investigation&rft.aulast=Casey%2C+E.&rft.au=Casey%2C+E.&rft.date=2019&rft.volume=31&rft.pages=200898&rft_id=info:doi\/10.1016%2Fj.fsidi.2019.200898&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaseyForensic20-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CaseyForensic20_30-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-CaseyForensic20_30-1\">30.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Casey, E. (2020). \"Standardization of forming and expressing preliminary evaluative opinions on digital evidence\". <i>Forensic Science International: Digital Investigation<\/i> <b>32<\/b>: 200888. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.fsidi.2019.200888\" target=\"_blank\">10.1016\/j.fsidi.2019.200888<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Standardization+of+forming+and+expressing+preliminary+evaluative+opinions+on+digital+evidence&rft.jtitle=Forensic+Science+International%3A+Digital+Investigation&rft.aulast=Casey%2C+E.&rft.au=Casey%2C+E.&rft.date=2020&rft.volume=32&rft.pages=200888&rft_id=info:doi\/10.1016%2Fj.fsidi.2019.200888&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISOIEC17025-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISOIEC17025_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/66912.html\" target=\"_blank\">\"ISO\/IEC 17025:2017 General requirements for the competence of testing and calibration laboratories\"<\/a>. International Organization for Standardization. November 2017<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/66912.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/66912.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO%2FIEC+17025%3A2017+General+requirements+for+the+competence+of+testing+and+calibration+laboratories&rft.atitle=&rft.date=November+2017&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F66912.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194110\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.655 seconds\nReal time usage: 1.397 seconds\nPreprocessor visited node count: 23147\/1000000\nPreprocessor generated node count: 36951\/1000000\nPost\u2010expand include size: 158316\/2097152 bytes\nTemplate argument size: 58101\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 472.509 1 - -total\n 85.66% 404.743 1 - Template:Reflist\n 71.96% 339.994 31 - Template:Citation\/core\n 52.36% 247.411 21 - Template:Cite_journal\n 18.05% 85.272 8 - Template:Cite_web\n 8.05% 38.044 1 - Template:Infobox_journal_article\n 7.66% 36.215 1 - Template:Infobox\n 5.66% 26.755 2 - Template:Cite_book\n 5.24% 24.741 80 - Template:Infobox\/row\n 4.67% 22.079 22 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12446-0!*!0!!en!5!* and timestamp 20210429194109 and revision id 42109\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Digital_transformation_risk_management_in_forensic_science_laboratories\">https:\/\/www.limswiki.org\/index.php\/Journal:Digital_transformation_risk_management_in_forensic_science_laboratories<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","b18b6bb1bbee57c49590e7b3ba53097d_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/6f\/Fig1_Casey_ForensicSciInt2020_316.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/69\/Fig2_Casey_ForensicSciInt2020_316.jpg"],"b18b6bb1bbee57c49590e7b3ba53097d_timestamp":1619725269,"00bcc6c235846f18221f6a618b37957c_type":"article","00bcc6c235846f18221f6a618b37957c_title":"Towards a risk catalog for data management plans (Weng and Thoben 2020)","00bcc6c235846f18221f6a618b37957c_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Towards_a_risk_catalog_for_data_management_plans","00bcc6c235846f18221f6a618b37957c_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Towards a risk catalog for data management plans\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nTowards a risk catalog for data management plansJournal\n \nInternational Journal of Digital CurationAuthor(s)\n \nWeng, Franziska; Thoben, StellaAuthor affiliation(s)\n \nKiel UniversityPrimary contact\n \nEmail: franziskaweng at web dot deYear published\n \n2020Volume and issue\n \n15(1)Page(s)\n \n18DOI\n \n10.2218\/ijdc.v15i1.697ISSN\n \n1746-8256Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/www.ijdc.net\/article\/view\/697Download\n \nhttp:\/\/www.ijdc.net\/article\/view\/697\/614 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Related work \n4 Methods \n5 Risks \n\n5.1 Legal risks \n\n5.1.1 Penalty for conducting unreported notifiable practices [RLEGU] \n5.1.2 Penalty for unpermitted usage of external data [RLEGE] \n5.1.3 Penalty for unpermitted usage of personal data [RLEGP] \n5.1.4 Penalty for conducting inadequate data protection practices [RLEGD] \n\n\n5.2 Privacy risks \n\n5.2.1 Loss of confidentiality through sending data to an unintended recipient [RPRIR] \n5.2.2 Loss of confidentiality through interception or eavesdropping of information [RPRII] \n5.2.3 Loss of confidentiality through loss or theft of portable storage media or devices [RPRIS] \n5.2.4 Loss of confidentiality through careless data handling by an external party [RPRIE] \n\n\n5.3 Technical risks \n\n5.3.1 Unavailability through data corruption [RTECC] \n5.3.2 Unavailability through data loss [RTECL] \n\n\n5.4 Science risks \n\n5.4.1 Poor knowledge discovery or reusability for stakeholders cannot find [RSCIF], access [RSCIA], integrate [RSCII], or reuse [RSCIR] the data \n\n\n5.5 Preservation risk \n\n5.5.1 Unsustainability in the long-term through unavailability or discontinuity of financial support [RPREU] \n\n\n\n\n6 Evaluation \n7 Evaluation results \n8 Conclusion \n9 Appendices \n\n9.1 Appendix A: Interpretation notes \n9.2 Appendix B: Risk mitigations \n\n\n10 Footnotes \n11 Acknowledgements \n\n11.1 Funding \n\n\n12 References \n13 Notes \n\n\n\nAbstract \nAlthough data management and its careful planning are not new topics, there is little published research on risk mitigation in data management plans (DMPs). We consider it a problem that DMPs do not include a structured approach for the identification or mitigation of risks, because it would instill confidence and trust in the data and its stewards, and foster the successful conduction of data-generating projects, which often are funded research projects. In this paper, we present a lightweight approach for identifying general risk in DMPs. We introduce an initial version of a generic risk catalog for funded research and similar projects. By analyzing a selection of 13 DMPs for projects from multiple disciplines published in the Research Ideas and Outcomes (RIO) journal, we demonstrate that our approach is applicable to DMPs and transferable to multiple institutional constellations. As a result, the effort for integrating risk management in data management planning can be reduced.\nKeywords: data management plan, data management, risk management, risk assessment, information security\n\nIntroduction \nUniversity of New Mexico's William Michener describes a data management plan (DMP) as \"a document that describes how you will treat your data during a project and what happens with the data after the project ends.\u201d[1] The Digital Curation Centre's (DCC) Martin Donnelly notes that DMPs \u201cserve to mitigate risks and help instill confidence and trust in the data and its stewards.\u201d[2] Sarah Jones, also of the DCC, adds that \u201cplanning for the effective creation, management, and sharing of your data enables you to get the most out of your research.\u201d[3] As such, the creation of a DMP should not only happen for obtaining a grant but also for successfully conducting the proposed project.\nAccording to ISO 31000[4], a risk is \u201can effect of uncertainty on objectives.\u201d Data management plans should help to decrease effects of uncertainty on project objectives. We consider it a problem that neither DMPs nor funders\u2019 DMP evaluation schemes include a structured approach for the identification or mitigation of risks, since this would foster the successful conduction of data-generating projects, which often are funded research projects. We believe our approach will help funders evaluate risks of proposed projects and hence the risks of their investment options.\nData management maturity models like the Data Management Maturity (DMM) model[5] or the Enterprise Information Management (EIM) maturity model[6] are primarily designed for enterprises and may not be feasible for higher education institutions (HEIs). A rigid model for HEIs to coordinate support of data management and sharing across a diverse range of actors and processes to deliver the necessary technological and human infrastructures \u201ccannot be prescribed since individual organizations and cultures occupy a spectrum of differences.\u201d[7] Also, there is a potential conflict between organizational demands and scientific freedom. The Charter of Fundamental Rights of the E.U. contains scientific freedom as a constitutional right, and researchers may view the imposition of specific data management processes as a restriction of their scientific freedom. On an even more international level, the UNESCO recommends that \u201ceach Member State should institute procedures adapted to its needs for ensuring that, in the performance of research and development, scientific researchers respect public accountability while at the same time enjoying the degree of autonomy appropriate to their task and to the advancement of science and technology.\u201d[8]\nWe consider it important, that researchers commit themselves to data management practices like e.g., ISO 31000. However, ISO 31000 defines the risk management process as a feedback loop to be conducted in organizations.[4] Projects tend to have a much more limited scope with regard to funding and duration than organizations. Therefore, we regard the ISO 31000 risk management process as too time-consuming and of limited suitability for funded research and similar projects.\nIn this paper, we propose a lightweight approach for the identification of general risks in DMPs. We introduce an initial version of a generic risk catalog for funded research and similar projects. By analyzing a selection of 13 DMPs for projects from multiple disciplines published in the Research Ideas and Outcomes (RIO) journal[9][10][11][12][13][14][15][16][17][18][19][20][21], we demonstrate that our approach is applicable and transferable to multiple institutional constellations. As a result, the effort for integrating risk management in data management planning can be reduced.\n\nRelated work \nJones et al. developed a guide for HEIs \u201cto help institutions understand the key aims and issues associated with planning and implementing research data management (RDM) services.\u201d[7] In this guide, the authors mention data management risks for HEIs. They note that While the upfront costs for cheap storage of active data \u201cmay be only a fraction of those quoted by central services, the risks of data loss and security breaches are significantly higher, potentially leading to far greater costs in the long term.\u201d[7] Additionally, there are \u201cpotential legal risks from using third-party services.\u201d[7] However, data selection counters the risks of \u201creputational damage from exposing dirty, confidential, or undocumented data that has been retained long after the researchers who created it have left.\u201d[7]\nThe OSCRP working group developed the OSCRP (Open Science Cyber Risk Profile), which \u201cis designed to help principal investigators (PI) and their supporting information technology (IT) professionals assess cybersecurity risks related to open science projects.\u201d[22] The OSCRP working group proposes that principal investigators examine risks, consequences and avenues of attack for each mission critical science asset on an inventory list, whereas assets include devices, systems, data, personnel, workflows, and other kinds of resources.[22] We regard this as a very detailed alternative to our approach, but FAIR Guiding Principles[23] and long-term preservation need to be added.\nIn 2014, Ferreira et al.[24] \u201cpropose an analysis process for eScience projects using a data management plan and ISO 31000 in order to create a risk management plan that can complement the data management plan.\u201d The authors describe an analytical process for creating a risk management plan and \u201cpresent the previous process\u2019 validation, based on the MetaGen-FRAME project.\u201d[24] Within this validation Ferreira et al. also identify a project\u2019s task-specific risks, e.g., \u201cR6: Loss of metadata, denying the representation of the output information to the user via Taverna.\u201d[24] This risk is tailored to the use of Taverna and hence may not be relevant for the majority of funded research and similar projects. There may be projects for which analyzing specific risks for all resources may be crucial. However, a detailed risk analysis may require a considerable amount of work.\n\nMethods \nWe propose a lightweight approach that can serve as a starting point to include risk management in research data management planning. It doesn\u2019t preclude detailed approaches like OSCRP[22] or ISO 31000.[4] Instead, we propose an approach which tries to reduce and maybe avoid the burden of a full risk management process like, e.g., ISO 31000. Our approach is based on a pre-tailored and extensible general risk catalog (Table 1) to lessen the effort required for risk management. We derived part of this risk catalog from 29 interviews with researchers from multiple disciplines[a], which we conducted as part of project SynFo: Creating synergies on the operational level of research data management.[25] One goal of project SynFo was the development of a transferable approach to improve research data management in multiple organizational constellations. In generalized content from the interviews, we identified risks entailed by interfaces of information, e.g., between researchers and data subjects or between researchers and external service providers. For the development of our approach, we also consulted the catalogs for threats and measures from the supplement of the \u201cIT-Grundschutz\u201d catalogs[26] by the German Federal Office for Information Security (BSI), the FAIR Guiding Principles[23], and the report and action plan from the European Commission expert group on FAIR data.[27]\n\n\n\n\n\n\n\nTable 1. General risk catalog\n\n\n\nRisk category\n\nRisk [CODE]\n\nPossible risk source\n\n\nLegal\n\nPenalty for conducting unreported notifiable practices [RLEGU]\n\nPhysical sample collection\n\n\nPenalty for unpermitted usage of external data [RLEGE]\n\nProcessing external data\n\n\nPenalty for unpermitted usage of personal data [RLEGP]\n\nProcessing personal data\n\n\nPenalty for conducting inadequate data protection practices [RLEGD]\n\nUsing an external service provider for processing personal data\n\n\nPrivacy\n\nLoss of confidentiality through sending data to an unintended recipient [RPRIR]\n\nCorrespondence\n\n\nLoss of confidentiality through interception or eavesdropping of information [RPRII]\n\nOnline data transmission\n\n\nLoss of confidentiality through loss or theft of portable storage media or devices [RPRIS]\n\nPortable storage media or devices\n\n\nLoss of confidentiality through careless data handling by an external party [RPRIE]\n\nSharing data with an external party without publication purposes\n\n\nTechnical\n\nUnavailability through data corruption [RTECC]\n\nData processing\n\n\nUnavailability through data loss [RTECL]\n\nData storage\n\n\nScience\n\nPoor knowledge discovery or reusability for stakeholders cannot find the data [RSCIF]\n\nSearchable information not planned\n\n\nPoor knowledge discovery or reusability for stakeholders cannot access the data [RSCIA]\n\nSharing location not planned\n\n\nPoor knowledge discovery or reusability for stakeholders cannot integrate the data [RSCII]\n\nFile format not planned\n\n\nPoor knowledge discovery or reusability for stakeholders cannot reuse the data [RSCIR]\n\nLicensing and context information not planned\n\n\nPreservation\n\nUnsustainability in the long-term through unavailability or discontinuity of financial support [RPREU]\n\nPreservation location not planned\n\n\n\nOur risk identification includes risks, their possible risk sources, mitigation approaches, and consequences. By analyzing occurrences and mitigations of risks from our catalog within a selection of 13 DMPs from multiple disciplines[b], published in the \u2018\u2019RIO\u2019\u2019 journal, we demonstrate that our lightweight approach is applicable to DMPs and transferable to multiple institutional constellations. We evaluate the occurrences of the 15 risks in our catalog by identifying possible risk sources in each of the selected DMPs and analyze the risk mitigations in accordance to what the authors wrote.\n\nRisks \nLegal risks \nA breach of a regulation like the General Data Protection Regulation (GDPR) or the Nagoya Protocol can result in high fines. At worst, compliance breaches can lead to reputational damages, legal disputes, and enormous cost.\n\nPenalty for conducting unreported notifiable practices [RLEGU] \nResearch may include reportable research practices like the collection of physical samples regulated by the Nagoya Protocol on Access to Genetic Resources and the Fair and Equitable Sharing of Benefits Arising from their Utilization, which was transposed into E.U. law by Regulation (EU) No 511\/2014. Under this regulation, there is a reporting obligation if the research on genetic resources is financially supported (Art. 7, Sec. 1) and if the research is in the final stage of development of a product that is based on the utilization of genetic resources (Art. 7, Sec.2).[28] Article 11 says that \u201cMember States shall lay down the rules on penalties applicable to infringements of Articles 4 and 7 and shall take all the measures necessary to ensure that they are applied.\u201d[28] The Nagoya Protocol \u201cand EU documents themselves give no guidance on penalties, each country has the liberty to determine these.\u201d[29] Consequences may be fines of up to EUR 810,000 or even imprisonment.[29] To avoid penalties, the parties should comply strictly with the rules. The Convention on Biological Diversity publishes a detailed list of parties to the Nagoya Protocol.[c]\n\nPenalty for unpermitted usage of external data [RLEGE] \nIn many countries, data by themselves do not have inherent legal protection. License contracts can reach various agreements concerning terms of use. Free licenses make (data) objects available for utilization to everyone, but usage can be restricted or conditioned. Creative Commons (CC) licenses and the GNU General Public License (GPL), which is specialized for free software, are widely used. Nonetheless, using CC licenses can lead to conflicting rights of third parties. Publicity, personality, and privacy rights \u201cnot held by the licensor are not affected and may still affect your desired use of a licensed work.\u201d[30] \u201cIf there are any third parties who may have publicity, privacy, or personality rights that apply, those rights are not affected by your application of a CC license, and a reuser must seek permission for relevant uses.\u201d[30] This example holds for pictures of persons. Also, the GNU GPL license imposes transitive obligations, e.g., \u201cderivative programs must also be subject to the same initial GPL conditions of ability to copy, modify, or redistribute.\u201d[31] To mitigate the risk of unpermitted usage of external data, it is recommended to abide by the license terms. In general, an overview about the data and the related licenses can be developed in the DMP or within the framework of a data policy.\n\nPenalty for unpermitted usage of personal data [RLEGP] \nIn the E.U., the GDPR governs the processing of personal data. Articles 6 and 7 of the GDPR regulate the lawfulness of processing and the conditions of consent. On an international level, the European Commission can conduct an assessment to \u201censure that the level of data protection in a third country or international organization is essentially equivalent to the one established by the E.U. legislation.\u201d[32] Canada (commercial organizations), Israel, Switzerland, Japan, and the U.S. (limited to the Privacy Shield Framework) offer an adequate level of data protection.[33] To avoid penalties, it is recommended to receive written consent forms from data subjects, including information about the purpose and procedures of data processing.\n\nPenalty for conducting inadequate data protection practices [RLEGD] \nArticle 5 of the GDPR enumerates principles related to processing of personal data: the principle of lawfulness, fairness and transparency, purpose limitation, data minimization, accuracy, storage limitation, integrity, and confidentiality, as well as accountability. According to Article 45 of the GDPR, \u201cA transfer of personal data to a third country or an international organization may take place where the Commission has decided that the third country, a territory or one or more specified sectors within that third country, or the international organization in question ensures an adequate level of protection. Such a transfer shall not require any specific authorization.\u201d[34] Countries without adequacy, which are not classified as safe third countries, can guarantee protection in other ways, for example by using appropriate safeguards (Art. 6, GDPR) or binding corporate rules (Art. 7, GDPR). To avoid penalties, it is recommended to abide by the applicable laws. In case of doubt, researchers can contact the appropriate (data protection) authorities.\n\nPrivacy risks \nA loss of confidentiality can have adverse effects on an organization like financial effects.[26] These effects may also apply to a researcher who additionally may want to keep research data confidential before scientific output is published, so that research data will not be subject to theft of work.\n\nLoss of confidentiality through sending data to an unintended recipient [RPRIR] \nCorrespondence has the intrinsic potential that a researcher transmits data to an unintended recipient. This may happen accidentally or as the result of a fraudulent attack like social engineering and leads to loss of confidentiality. \u201cSocial engineering is a method used to gain unauthorized access to information or IT systems by social action.\u201d[26] Researchers should take extra care when sending confidential information and be aware of fraudulent attacks.\n\nLoss of confidentiality through interception or eavesdropping of information [RPRII] \nIn the supplement of the IT-Grundschutz catalogs, the BSI specifies the threats of interception or eavesdropping of information, which entail the risk of loss of confidentiality.[26] \u201cSince data is sent using unforeseeable routes and nodes on the internet, the sent data should only be transmitted in an encrypted form, as far as possible.\u201d[26]\n\nLoss of confidentiality through loss or theft of portable storage media or devices [RPRIS] \n\u201cPortable terminal devices and mobile data media in particular can be lost easily\u201d[26] or even be stolen. \u201cWhenever possible, mobile data media such as USB sticks and laptops should always be encrypted completely even if they are only occasionally used for confidential information.\u201d[26]\n\nLoss of confidentiality through careless data handling by an external party [RPRIE] \nWe regard the event that researchers share data with an external party without the purpose of publication as entailing the risk of loss of confidentiality. The external party may handle confidential data carelessly. \u201cIt can frequently be observed that there are a number of organizational or technical security procedures available in organizations, but they are then undermined through careless handling of the specifications and the technology.\u201d[26] We recommend that researchers who share their research data to always grant specific usage rights in written form to the external party or to check if appropriate security measures are applied by the external party.\n\nTechnical risks \nData can lose their integrity or be lost[26], leading to the major risk of unavailability of data. Unavailability of the correct data through silent corruption can lead to usage of incorrect data and hence to the production of incorrect results. If data are unavailable, either the project may fail or researchers need to repeat their data collection and the project will be behind schedule.\n\nUnavailability through data corruption [RTECC] \n\u201cThe integrity of information may be impaired due to different causes, e.g., manipulations, errors caused by people, incorrect use of applications, malfunctions of software, or transmission errors.\u201d[26] \u201cIf only accidental changes need to be detected, then checksum procedures (e.g., cyclic redundancy checks) or error-correcting codes can be used.\u201d[26] Nonetheless, there may be other scenarios where these verification techniques are insufficient.\n\nUnavailability through data loss [RTECL] \nData may \u201cbe lost when devices or data media are damaged, lost, or stolen,\u201d[26] hence becoming unavailable. Approaches to mitigate irretrievable losses of data are for example regular backups[26] or keeping copies in multiple storage locations.[35]\n\nScience risks \nConsequences of poor discoverability and reusability of data are that researchers may unnecessarily repeat work and that scientific outputs derived from it may fail to be comprehensible, reproducible, or traceable. Problems with reproducibility and replication \u201ccan cause permanent damage to the credibility of science,\u201d[36] and thus why this category is referred to as \"science risks.\"\n\nPoor knowledge discovery or reusability for stakeholders cannot find [RSCIF], access [RSCIA], integrate [RSCII], or reuse [RSCIR] the data \nMaking data fundable, accessible, interoperable, and reusable (FAIR) to human and computational stakeholders is a best practice approach described in the FAIR Guiding Principles of Wilkinson et al.[23] Therefore, we include the risks that stakeholders cannot find, access, process, or reuse data in our risk catalog. Authors of DMPs can mitigate these risks as described by Wilkinson et al.[23] We abbreviated the risk names under this risk category using the term \u201cpoor knowledge discovery or reusability,\u201d but refer to all the FAIR Guiding Principles for more information.[23]\n\nPreservation risk \nIf data are not suitably preserved, scientific outputs derived from them may fail to be comprehensible, reproducible, or traceable in the long run. Data should be stored in a trusted and sustainable digital repository.[27]\n\nUnsustainability in the long-term through unavailability or discontinuity of financial support [RPREU] \nA digital preservation location has the intrinsic technical risk that data become unavailable through data loss or corruption. However, preservation locations also entail the risk of becoming unavailable when their funding ends. For example, Canhos states that discontinuity of financial support is a threat to Brazil\u2019s Virtual Herbarium and its data sources.[9] Authors of DMPs should consider these risks when selecting a preservation location. They can mitigate the risk that data are not preserved long-term by reviewing the external preservation location\u2019s longevity, certificates, and funding. We also suggest that attention is paid to possible migration and exit strategies like exporting and handing over data to a national data archive. This may particularly be important when the preservation location is not external.\n\nEvaluation \nWhen applying the risk catalog (Table 1) to the sample of 13 DMPs, we distinguish between risk occurrences themselves and risk occurrences with at least one mitigation, as show in Table 2.\n\n\n\n\n\n\n\nTable 2. Risk occurrences (+) and risk occurrences with at least one mitigation (-) in the sample of 13 DMPs\n\n\n\nDMP\n\nRLEGU\n\nRLEGE\n\nRLEGP\n\nRLEGD\n\nRPRIR\n\nRPRII\n\nRPRIS\n\nRPRIE\n\nRTECC\n\nRTECL\n\nRSCIF\n\nRSCIA\n\nRSCII\n\nRSCIR\n\nRPREU\n\n\nCanhos, 2017[9]\n\n\n\n-\n\n\n\n\n\n\n\n\n\n\n\n\n\n+\n\n-\n\n-\n\n-\n\n+\n\n-\n\n+\n\n\nFey and Anderson, 2016[10]\n\n\n\n+\n\n\n\n\n\n\n\n\n\n+\n\n\n\n+\n\n-\n\n-\n\n-\n\n-\n\n-\n\n-\n\n\nFisher and Nading, 2016[11]\n\n\n\n+\n\n+\n\n+\n\n\n\n+\n\n\n\n+\n\n+\n\n-\n\n-\n\n-\n\n-\n\n+\n\n-\n\n\nGatto, 2017[12]\n\n\n\n+\n\n\n\n\n\n\n\n\n\n\n\n\n\n+\n\n-\n\n-\n\n-\n\n+\n\n-\n\n-\n\n\nMcWhorter et al., 2016[13]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n-\n\n-\n\n+\n\n-\n\n-\n\n-\n\n-\n\n\nNeylon, 2017[14]\n\n\n\n\n\n+\n\n+\n\n+\n\n+\n\n\n\n+\n\n+\n\n-\n\n-\n\n-\n\n-\n\n-\n\n-\n\n\nNichols and Stolze, 2016[15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n-\n\n-\n\n-\n\n-\n\n-\n\n-\n\n-\n\n\nPannell, 2016[16]\n\n+\n\n+\n\n\n\n\n\n\n\n+\n\n\n\n-\n\n+\n\n-\n\n-\n\n-\n\n-\n\n-\n\n-\n\n\nTraynor, 2017[17]\n\n\n\n\n\n-\n\n+\n\n+\n\n+\n\n+\n\n+\n\n+\n\n-\n\n+\n\n-\n\n-\n\n-\n\n-\n\n\nWael, 2017[18]\n\n\n\n+\n\n+\n\n+\n\n+\n\n+\n\n+\n\n-\n\n+\n\n-\n\n+\n\n-\n\n+\n\n-\n\n+\n\n\nWhite, 2016[19]\n\n\n\n-\n\n\n\n\n\n\n\n\n\n\n\n\n\n+\n\n-\n\n+\n\n-\n\n+\n\n-\n\n+\n\n\nWoolfrey, 2017[20]\n\n\n\n-\n\n\n\n\n\n\n\n\n\n\n\n\n\n+\n\n-\n\n+\n\n-\n\n-\n\n-\n\n-\n\n\nXu et al., 2016[21]\n\n+\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n+\n\n-\n\n-\n\n-\n\n+\n\n-\n\n-\n\n\n\nBecause risk sources and mitigations were not always explicitly mentioned in the 13 sample DMPs, we needed to make interpretations. Appendix A shows our interpretation notes. According to these interpretations, we then found the mitigations, as shown in Appendix B.\n\nEvaluation results \nEach of the 15 risks of our catalog occurred in at least two of the selected 13 DMPs. Table 3 summarizes our evaluation results.\n\n\n\n\n\n\n\nTable 3. Summary of risk evaluation results\n\n\n\nRisk from catalog\n\n% of risk occurrences mitigated\n\nMost often used mitigation strategy (in # of DMPs)\n\n\nUnavailability through data loss [RTECL]\n\n100.00\n\nBackup (8)\n\n\nPoor knowledge discovery or reusability for stakeholders cannot access the data [RSCIA]\n\n100.00\n\nSpecific repository (7)\n\n\nPoor knowledge discovery or reusability for stakeholders cannot reuse the data [RSCIR]\n\n92.31\n\nSpecific license (9)\n\n\nUnsustainability in the long-term through unavailability or discontinuity of financial support [RPREU]\n\n76.92\n\nSpecific file formats (4), Specific data archive (4)\n\n\nPoor knowledge discovery or reusability for stakeholders cannot integrate the data [RSCII]\n\n69.23\n\nSpecific file formats (8)\n\n\nPoor knowledge discovery or reusability for stakeholders cannot find the data [RSCIF]\n\n61.54\n\nMetadata (2)\n\n\nLoss of confidentiality through careless data handling by an external party [RPRIE]\n\n40.00\n\nAgreement for IP rights (1), Secure external infrastructure (1)\n\n\nPenalty for unpermitted usage of external data [RLEGE]\n\n37.50\n\nRespect usage permissions of external data (2)\n\n\nPenalty for unpermitted usage of personal data [RLEGP]\n\n25.00\n\nSigned consent forms (1)\n\n\nUnavailability through data corruption [RTECC]\n\n15.38\n\nCompare data from before and after transmission (1), Data quality control (1)\n\n\nLoss of confidentiality through interception or eavesdropping of information [RPRII]\n\n00.00\n\n\n\n\nPenalty for conducting inadequate data protection practices [RLEGD]\n\n00.00\n\n\n\n\nLoss of confidentiality through sending data to an unintended recipient [RPRIR]\n\n00.00\n\n\n\n\nLoss of confidentiality through loss or theft of portable storage media or devices [RPRIS]\n\n00.00\n\n\n\n\nPenalty for conducting unreported notifiable practices [RLEGU]\n\n00.00\n\n\n\n\n\nWithin the small sample of 13 DMPs, we found 34 distinct strategies to mitigate 10 of the 15 risks of our proposed catalog. Hence, we also found that for five of the 15 risks from our catalog the authors did not describe any mitigation in the corresponding DMP. These risks are legal and privacy risks and they do have possible consequences like loss of reputation or project failure through theft of work. The authors of the selected DMPs overall attach highest importance to mitigating data unavailability through data loss; making data findable, accessible, interoperable, and reusable; and ensuring their long-term digital preservation. We found that all the authors of the 13 DMPs managed to mitigate two risks from our catalog: unavailability through data loss (RTECL) and poor knowledge discoverability or reusability for stakeholders cannot access the data (RSCIA).\n\nConclusion \nSince we identified each risk of our catalog in at least two of the selected DMPs, we conclude that our risk catalog is applicable to DMPs from multiple areas of research. In the selected DMPs, we overall find 53 of 125 (42.4%) risk occurrences not mitigated and hence see the necessity of DMP quality improvement through risk identification and mitigation planning in the data management planning phase.\nWe consider our approach useful for identifying general risks in DMPs. We propose that after filling out a funder\u2019s DMP template, authors of DMPs refer to a risk catalog to identify possible risk sources and hence risks. Next, the authors should add mitigations to their DMP in the corresponding paragraph, if their DMP does not already contain one. For example, in a DMP\u2019s paragraph in which authors write about the usage of external hard disks, they should add a sentence indicating that these external hard disks will be encrypted to mitigate the risk of loss of confidentiality through loss or theft of storage media, if their DMP does not yet contain any measures mitigating this risk.\nThe risk catalog may also be useful to funders, since it makes it possible for them to evaluate basic investment risks of proposed projects.\nNote that many of the legal assertions in this article hold within the E.U. Applicability to non-E.U. countries may vary.\nWe think further research on suitable risk management approaches concerning the data management of funded research and similar projects needs to be conducted.\n\nAppendices \nAppendix A: Interpretation notes \nCanhos states that discontinuity of financial support is a threat to Brazil\u2019s Virtual Herbarium and its data sources.[9] We interpret this as the risk that data are not preserved long-term.\nIn Fey and Anderson's DMP[10], we interpret the choice of .txt and .csv formats as open file formats for interoperability, and we interpret the use of metadata standards as to mitigate the risk that data are not findable.\nIn Fisher and Nading\u2019s DMP [11], we find only geospatial metadata mentioned. Neither documentation nor license information are mentioned in Fisher and Nading\u2019s DMP. [11]\nIn Gatto\u2019s DMP[12], we assume data are enriched or combined so that a license of the resulting data set should be derived from the source data licenses in Gatto\u2019s DMP. We evaluate Gatto\u2019s DMP according to the FAIR Guiding Principles for research software, as proposed by Jim\u00e9nez et al.[37]\nConcerning the DMP of McWhorter et al.[13], we assume that data do not have protection requirements and no risk of interception because data are made freely available for public use.\nNeylon\u2019s DMP[14] does not include collecting signed forms of consent from interviewees. Neylon also decides not to make all data anonymous and accessible.[14] Finally, the DMP does not include using file formats that are interoperable or allow re-use.\nNichols and Stolze\u2019s DMP[15] describes migration of data from old storage media to an .xlsx format and their subsequent publication.\nConcerning Pannell\u2019s DMP[16], we think that it would be adequate to inform the responsible authority about the planned research project. Pannell does not address rights of use of external data. We interpret Pannell's term \"filterable,\"[16] in the context of metadata documentation, as meaning \"findable.\"\nIn Traynor\u2019s DMP[17], it is not clear if personal data are anonymized before they are uploaded in the infrastructure of an external service provider. Traynor\u2019s DMP also contains no decisions for metadata capture or a specific long-term preservation location.[17]\nAs for Wael\u2019s DMP, they plan to hire a consultant to do technical planning and system set up.[18] The DMP does not include making data interoperable. We think that utilizing academic contacts to inform the research community that the data exists, as stated by Wael[18], is not the same as making data findable.\nAccording to White\u2019s DMP, the project members will develop data and software \u201cin the open,\u201d[19] which we interpret as making data accessible. In his DMP, White does not mention long-term preservation.[19] We regard the metadata capture and user-focused documentation stated in White\u2019s DMP[19] as making data reusable.\nIn Woolfrey\u2019s DMP[20], metadata are captured for re-usability. Woolfrey\u2019s DMP does not include making data findable.\nXu et al.[21] do not explicitly state in which states or countries they plan to collect physical samples. We make the interpretation that physical samples are registered with a persistent identifier, as described by Xu et al.[21], to make their metadata findable. Xu et al. write that for re-use and distribution, IEDA (Interdisciplinary Earth Data Alliance) would have a persistent identifier assigned to the data sets.[21]\n\nAppendix B: Risk mitigations \n\n\n\n\n\n\nTable 4. Risk mitigations\n\n\n\nDMP\n\nRisk mitigations\n\n\nCanhos, 2017[9]\n\nAligned licensing of all data (RLEGE), Backup (RTECL), Maintain blog and social media account (RSCIF), Publicly accessible server (RSCIA), Specific fle formats (RSCIR), Specific license (RSCIR), Metadata or citation of external data (RSCIR), Standard data model (RSCIR)\n\n\nFey and Anderson, 2016[10]\n\nBackup (RTECL), Provide rights of use (RSCIR); Specific data archive (RPREU, RSCIA), Metadata (RSCIF), Metadata standard (RSCIF), Publicly accessible server (RSCIA), Specific file formats (RSCII)\n\n\nFisher and Nading, 2016[11]\n\nBackup (RTECL), Specific data archive (RPREU, RSCIA), Listing in national discipline-specific wiki (RSCIF), Listing on funders website (RSCIF), Specific file formats (RSCII)\n\n\nGatto, 2017[12]\n\nMultiple storage locations (RTECL), Specific license (RSCIR), Collaborative software development (RPREU, RSCIF), Specific repository (RSCIA), Documentation (RSCIR), Metadata or citation of external data (RSCIR)\n\n\nMcWhorter et al., 2016[13]\n\nData are freely available for public use (RSCIR), Data quality control (RTECC), Backup (RTECL), Multiple preservation locations (RTECL, RPREU), Specific data archive (RPREU), Publicly accessible server (RSCIA), Specific file formats (RSCII), Metadata (RSCIR)\n\n\nNeylon, 2017[14]\n\nMultiple storage locations (RTECL), Specific license (RSCIR), Multiple preservation locations (RPREU), Specific file formats (RPREU, RSCII), Persistent identifier (RSCIF), Specific repository (RSCIA), Documentation (RSCIR)\n\n\nNichols and Stolze, 2016[15]\n\nCompare data from before and after transmission (RTECC), Backup (RTECL), Multiple storage locations (RTECL), Specific license (RSCIR), Repository guarantees long-term availability (RPREU), Publish data descriptor in open-access journal (RSCIF, RSCIR), Specific repository (RSCIA), Specific file formats (RSCII), Documentation (RSCIR), Metadata (RSCIR), Persistent identifier (RSCIR)\n\n\nPannell, 2016[16]\n\nSecure external service infrastructure (RPRIE), Replicas in external service infrastructure (RTECL), Specific license (RSCIR), Preservation at institution\u2019s library (RPREU), Specific file formats (RPREU, RSCII), Specific repository (RSCIA), Metadata (RSCIF), Persistent identifier (RSCIR)\n\n\nTraynor, 2017[17]\n\nSigned consent forms (RLEGP), Multiple storage locations (RTECL), Backup (RTECL), Specific license (RSCIR), Specific file formats (RPREU), Specific repository (RSCIA), Specific file formats (RSCII), Documentation (RSCIR)\n\n\nWael, 2017[18]\n\nMultiple storage locations (RTECL), Agreement for IP rights (RPRIE), Specific license (RSCIR), Publicly accessible server (RSCIA), Anonymization (RSCIR), Documentation (RSCIR), Specific file formats (RSCIR)\n\n\nWhite, 2016[19]\n\nRespect usage permissions of external data (RLEGE), Backup (RTECL), Specific license (RSCIR), Specific repository (RSCIA), Metadata (RSCIR), Documentation (RSCIR)\n\n\nWoolfrey, 2017[20]\n\nRespect usage permissions of external data (RLEGE), Backup (RTECL), Data are freely available for public use (RSCIR), Specific data archive (RPREU, RSCIA), Specific file formats (RSCII), Documentation (RSCIR), Metadata (RSCIR), Metadata standard (RSCIR)\n\n\nXu et al., 2016[21]\n\nMultiple storage locations (RTECL), Specific license (RSCIR), Specific file formats (RPREU), Specific repository (RPREU, RSCIA), Persistent identifier for physical samples (RSCIF), Documentation (RSCIR), Metadata (RSCIR); Persistent identifier (RSCIR), Standardized vocabulary (RSCIR)\n\n\n\nFootnotes \n\n\n\u2191 Geo sciences (12), biology (5), humanities (5), social and behavioral sciences (4), computer science, systems engineering and electrical engineering (2), and medicine (1) \n\n\u2191 Biology (4), geo sciences (4), social and behavioral sciences (3), computer science, systems engineering and electrical engineering (1), and humanities (1) \n\n\u2191 Parties to the Nagoya Protocol \n\n\nAcknowledgements \nWe thank Peter Wullinger and Klaus Stein for their constructive feedback.\n\nFunding \nThis work was funded by the German Federal Ministry of Education and Research (BMBF) under the funding ID 16FDM024.\n\nReferences \n\n\n\u2191 Michener, W.K. (2015). \"Ten Simple Rules for Creating a Good Data Management Plan\". PLoS Computational Biology 11 (10): e1004525. doi:10.1371\/journal.pcbi.1004525.   \n\n\u2191 Donnelly, M. (2012). \"Chapter 5: Data management plans and planning\". In Pryor, G.. Managing Research Data. Facet. pp. 83\u2013104. doi:10.29085\/9781856048910.006. ISBN 9781856048910.   \n\n\u2191 Jones, S. (2011). \"How to Develop a Data Management and Sharing Plan\". Digital Curation Centre. https:\/\/www.dcc.ac.uk\/guidance\/how-guides\/develop-data-plan . Retrieved 19 November 2019 .   \n\n\u2191 4.0 4.1 4.2 \"ISO 31000:2018 Risk management \u2014 Guidelines\". International Organization for Standardization. February 2018. https:\/\/www.iso.org\/standard\/65694.html .   \n\n\u2191 \"Data Management Maturity (DMM)\". Information System Audit and Control Association, Inc. 2019. https:\/\/cmmiinstitute.com\/data-management-maturity . Retrieved 22 November 2019 .   \n\n\u2191 Newman, D.; Logan, D. (23 December 2008). \"Overview: Gartner Introduces the EIM Maturity Model\". Gartner. https:\/\/www.gartner.com\/en\/documents\/846312\/overview-gartner-introduces-the-eim-maturity-model .   \n\n\u2191 7.0 7.1 7.2 7.3 7.4 Jones, S.; Pryor, G.; Whyte, A. (25 March 2013). \"How to Develop RDM Services - A guide for HEIs\". Digital Curation Centre. https:\/\/www.dcc.ac.uk\/guidance\/how-guides\/how-develop-rdm-services . Retrieved 19 November 2019 .   \n\n\u2191 UNESCO (2017). \"Records of the General Conference, 39th session, Paris, 30 October-14 November 2017, v. 1: Resolutions\". p. 116. https:\/\/unesdoc.unesco.org\/ark:\/48223\/pf0000260889.page=116 . Retrieved 30 November 2019 .   \n\n\u2191 9.0 9.1 9.2 9.3 9.4 Canhos, D.A.L. (2017). \"Data Management Plan: Brazil's Virtual Herbarium\". RIO 3: e14675. doi:10.3897\/rio.3.e14675.   \n\n\u2191 10.0 10.1 10.2 10.3 Fey, J.; Anderson, S. (2016). \"Boulder Creek Critical Zone Observatory Data Management Plan\". RIO 2: e9419. doi:https:\/\/doi.org\/10.3897\/rio.2.e9419.   \n\n\u2191 11.0 11.1 11.2 11.3 11.4 Fisher, J.; Nading, A.M. (2016). \"A Political Ecology of Value: A Cohort-Based Ethnography of the Environmental Turn in Nicaraguan Urban Social Policy\". RIO 2: e8720. doi:10.3897\/rio.2.e8720.   \n\n\u2191 12.0 12.1 12.2 12.3 Gatto, L. (2017). \"Data Management Plan for a Biotechnology and Biological Sciences Research Council (BBSRC) Tools and Resources Development Fund (TRDF) Grant\". RIO 3: e11624. doi:10.3897\/rio.3.e11624.   \n\n\u2191 13.0 13.1 13.2 13.3 McWhorter, J.; Wright, D.; Thomas, J. (2016). \"Coastal Data Information Program (CDIP)\". RIO 2: e8827. doi:10.3897\/rio.2.e8827.   \n\n\u2191 14.0 14.1 14.2 14.3 14.4 Neylon, C. (2017). \"Data Management Plan: IDRC Data Sharing Pilot Project\". RIO 3: e14672. doi:10.3897\/rio.3.e14672.   \n\n\u2191 15.0 15.1 15.2 15.3 Nichols, H.; Stolze, S. (2016). \"Migration of legacy data to new media formats for long-time storage and maximum visibility: Modern pollen data from the Canadian Arctic (1972\/1973)\". RIO 2: e10269. doi:10.3897\/rio.2.e10269.   \n\n\u2191 16.0 16.1 16.2 16.3 16.4 Pannell, J.L. (2016). \"Data Management Plan for PhD Thesis \"Climatic Limitation of Alien Weeds in New Zealand: Enhancing Species Distribution Models with Field Data\"\". RIO 2: e10600. doi:10.3897\/rio.2.e10600.   \n\n\u2191 17.0 17.1 17.2 17.3 17.4 Traynor, C. (2017). \"Data Management Plan: Empowering Indigenous Peoples and Knowledge Systems Related to Climate Change and Intellectual Property Rights\". RIO 3: e15111. doi:10.3897\/rio.3.e15111.   \n\n\u2191 18.0 18.1 18.2 18.3 18.4 Wael, R. (2017). \"Data Management Plan: HarassMap\". RIO 3: e15133. doi:10.3897\/rio.3.e15133.   \n\n\u2191 19.0 19.1 19.2 19.3 19.4 19.5 White, E.P. (2016). \"Data Management Plan for Moore Investigator in Data Driven Discovery Grant\". RIO 2: e10708. doi:10.3897\/rio.2.e10708.   \n\n\u2191 20.0 20.1 20.2 20.3 Woolfrey, L. (2017). \"Data Management Plan: Opening access to economic data to prevent tobacco related diseases in Africa\". RIO 3: e14837. doi:10.3897\/rio.3.e14837.   \n\n\u2191 21.0 21.1 21.2 21.3 21.4 21.5 Xu, H.; Ishida, M.; Wang, M. (2016). \"A Data Management Plan for Effects of particle size on physical and chemical properties of mine wastes\". RIO 2: e11065. doi:10.3897\/rio.2.e11065.   \n\n\u2191 22.0 22.1 22.2 Peisert, S.; Welch, V.; Adams, A. et al. (2017). \"Open Science Cyber Risk Profile (OSCRP)\". IUScholar Works. http:\/\/hdl.handle.net\/2022\/21259 . Retrieved 19 November 2019 .   \n\n\u2191 23.0 23.1 23.2 23.3 23.4 Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J. et al. (2016). \"The FAIR Guiding Principles for scientific data management and stewardship\". Scientific Data 3: 160018. doi:10.1038\/sdata.2016.18. PMC PMC4792175. PMID 26978244. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175 .   \n\n\u2191 24.0 24.1 24.2 Ferreira, F.; Coimbra, M.E.; Bairr\u00e3o, R. et al. (2014). \"Data Management in Metagenomics: A Risk Management Approach\". International Journal of Digital Curation 9 (1): 41\u201356. doi:10.2218\/ijdc.v9i1.299.   \n\n\u2191 University Computing Centre (Rechenzentrum) (2019). \"SynFo - Creating synergies on the operational level of research data management\". Kiel University. https:\/\/www.rz.uni-kiel.de\/en\/projects\/synfo-creating-synergies-on-the-operational-level-of-research-data-management .   \n\n\u2191 26.00 26.01 26.02 26.03 26.04 26.05 26.06 26.07 26.08 26.09 26.10 26.11 26.12 German Federal Office for Information Security (22 December 2016). \"IT-Grundschutz-catalogues 15th version - 2015 (Draft)\". Archived from the original on 28 January 2020. https:\/\/web.archive.org\/web\/20200128211607\/https:\/\/www.bsi.bund.de\/SharedDocs\/Downloads\/DE\/BSI\/Grundschutz\/International\/GSK_15_EL_EN_Draft.html . Retrieved 19 November 2019 .   \n\n\u2191 27.0 27.1 Collins, S.; Genova, F.; Harrower, N. (26 November 2018). \"Turning FAIR into reality\". European Commission. doi:10.2777\/1524. https:\/\/op.europa.eu\/en\/publication-detail\/-\/publication\/7769a148-f1f6-11e8-9982-01aa75ed71a1\/language-en .   \n\n\u2191 28.0 28.1 \"Regulation (EU) No 511\/2014 of the European Parliament and of the Council of 16 April 2014 on compliance measures for users from the Nagoya Protocol on Access to Genetic Resources and the Fair and Equitable Sharing of Benefits Arising from their Utilization in the Union Text with EEA relevance\". EUR-Lex. European Union. 16 April 2014. http:\/\/data.europa.eu\/eli\/reg\/2014\/511\/oj . Retrieved 01 December 2019 .   \n\n\u2191 29.0 29.1 \"Implementation of Nagoya Protocol: A comparison between The Netherlands, Belgium and Germany\". vo.eu. 18 June 2018. https:\/\/publications.vo.eu\/implementation-of-nagoya-protocol\/ . Retrieved 30 November 2019 .   \n\n\u2191 30.0 30.1 \"Frequently Asked Questions\". Creative Commons. https:\/\/creativecommons.org\/faq\/ . Retrieved 08 December 2019 .   \n\n\u2191 Lipinski, T.A. (2012). Librarian's Legal Companion for Licensing Information Resources and Legal Services. Neal-Schuman Publishers. p. 312. ISBN 9781555706104.   \n\n\u2191 Article 29 Data Protection Working Party (06 February 2018). \"Working document on Adequacy Referential (wp254rev.01)\". European Commission. https:\/\/ec.europa.eu\/newsroom\/article29\/item-detail.cfm?item_id=614108 . Retrieved 01 December 2019 .   \n\n\u2191 European Commission (14 January 2019). \"Adequacy decisions\". European Commission. https:\/\/ec.europa.eu\/info\/law\/law-topic\/data-protection\/international-dimension-data-protection\/adequacy-decisions_en . Retrieved 01 December 2019 .   \n\n\u2191 \"Consolidated text: Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation) (Text with EEA relevance)\". EUR-Lex. European Union. 27 April 2016. http:\/\/data.europa.eu\/eli\/reg\/2016\/679\/2016-05-04 . Retrieved 30 November 2019 .   \n\n\u2191 Reich, V.; Rosenthal, D.S.H. (2009). \"LOCKSS (Lots of Copies Keep Stuff Safe)\". New Review of Academic Librarianship 6 (1): 155\u201361. doi:10.1080\/13614530009516806.   \n\n\u2191 Peng, R. (2015). \"The reproducibility crisis in science: A statistical counterattack\". Sginificance 12 (3): 30\u201332. doi:10.1111\/j.1740-9713.2015.00827.x.   \n\n\u2191 Jim\u00e9nez, R.C.; Kuzak, M; Alhamdoosh, M. et al. (2017). \"Four simple recommendations to encourage best practices in research software [version 1; peer review: 3 approved]\". F1000Research 6: 876. doi:10.12688\/f1000research.11407.1.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; this version lists them in order of appearance, by design.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Towards_a_risk_catalog_for_data_management_plans\">https:\/\/www.limswiki.org\/index.php\/Journal:Towards_a_risk_catalog_for_data_management_plans<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on cybersecurityLIMSwiki journal articles on data management and sharing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 6 March 2021, at 19:34.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 438 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","00bcc6c235846f18221f6a618b37957c_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Towards_a_risk_catalog_for_data_management_plans skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Towards a risk catalog for data management plans<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Although <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a> and its careful planning are not new topics, there is little published research on <a href=\"https:\/\/www.limswiki.org\/index.php\/Risk_management\" title=\"Risk management\" class=\"wiki-link\" data-key=\"ecfbfb2550c23f053bd1aa0d9ea63e4e\">risk mitigation<\/a> in data management plans (DMPs). We consider it a problem that DMPs do not include a structured approach for the <a href=\"https:\/\/www.limswiki.org\/index.php\/Risk_assessment\" title=\"Risk assessment\" class=\"wiki-link\" data-key=\"e06196a3e3b70b62af61a54e93579f37\">identification<\/a> or mitigation of risks, because it would instill confidence and trust in the data and its stewards, and foster the successful conduction of data-generating projects, which often are funded research projects. In this paper, we present a lightweight approach for identifying general risk in DMPs. We introduce an initial version of a generic risk catalog for funded research and similar projects. By analyzing a selection of 13 DMPs for projects from multiple disciplines published in the <i>Research Ideas and Outcomes<\/i> (<i>RIO<\/i>) journal, we demonstrate that our approach is applicable to DMPs and transferable to multiple institutional constellations. As a result, the effort for integrating risk management in data management planning can be reduced.\n<\/p><p><b>Keywords<\/b>: data management plan, data management, risk management, risk assessment, information security\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>University of New Mexico's William Michener describes a data management plan (DMP) as \"a document that describes how you will treat your data during a project and what happens with the data after the project ends.\u201d<sup id=\"rdp-ebb-cite_ref-MichenerTen15_1-0\" class=\"reference\"><a href=\"#cite_note-MichenerTen15-1\">[1]<\/a><\/sup> The Digital Curation Centre's (DCC) Martin Donnelly notes that DMPs \u201cserve to mitigate risks and help instill confidence and trust in the data and its stewards.\u201d<sup id=\"rdp-ebb-cite_ref-DonnellyData12_2-0\" class=\"reference\"><a href=\"#cite_note-DonnellyData12-2\">[2]<\/a><\/sup> Sarah Jones, also of the DCC, adds that \u201cplanning for the effective creation, management, and sharing of your data enables you to get the most out of your research.\u201d<sup id=\"rdp-ebb-cite_ref-JonesHowTo11_3-0\" class=\"reference\"><a href=\"#cite_note-JonesHowTo11-3\">[3]<\/a><\/sup> As such, the creation of a DMP should not only happen for obtaining a grant but also for successfully conducting the proposed project.\n<\/p><p>According to ISO 31000<sup id=\"rdp-ebb-cite_ref-ISO31000_4-0\" class=\"reference\"><a href=\"#cite_note-ISO31000-4\">[4]<\/a><\/sup>, a risk is \u201can effect of uncertainty on objectives.\u201d Data management plans should help to decrease effects of uncertainty on project objectives. We consider it a problem that neither DMPs nor funders\u2019 DMP evaluation schemes include a structured approach for the <a href=\"https:\/\/www.limswiki.org\/index.php\/Risk_assessment\" title=\"Risk assessment\" class=\"wiki-link\" data-key=\"e06196a3e3b70b62af61a54e93579f37\">identification<\/a> or <a href=\"https:\/\/www.limswiki.org\/index.php\/Risk_management\" title=\"Risk management\" class=\"wiki-link\" data-key=\"ecfbfb2550c23f053bd1aa0d9ea63e4e\">mitigation<\/a> of risks, since this would foster the successful conduction of data-generating projects, which often are funded research projects. We believe our approach will help funders evaluate risks of proposed projects and hence the risks of their investment options.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">Data management<\/a> maturity models like the Data Management Maturity (DMM) model<sup id=\"rdp-ebb-cite_ref-ISACA_DMM_5-0\" class=\"reference\"><a href=\"#cite_note-ISACA_DMM-5\">[5]<\/a><\/sup> or the Enterprise Information Management (EIM) maturity model<sup id=\"rdp-ebb-cite_ref-NewmanOver08_6-0\" class=\"reference\"><a href=\"#cite_note-NewmanOver08-6\">[6]<\/a><\/sup> are primarily designed for enterprises and may not be feasible for higher education institutions (HEIs). A rigid model for HEIs to coordinate support of data management and sharing across a diverse range of actors and processes to deliver the necessary technological and human infrastructures \u201ccannot be prescribed since individual organizations and cultures occupy a spectrum of differences.\u201d<sup id=\"rdp-ebb-cite_ref-JonesHowTo13_7-0\" class=\"reference\"><a href=\"#cite_note-JonesHowTo13-7\">[7]<\/a><\/sup> Also, there is a potential conflict between organizational demands and scientific freedom. The Charter of Fundamental Rights of the E.U. contains scientific freedom as a constitutional right, and researchers may view the imposition of specific data management processes as a restriction of their scientific freedom. On an even more international level, the UNESCO recommends that \u201ceach Member State should institute procedures adapted to its needs for ensuring that, in the performance of research and development, scientific researchers respect public accountability while at the same time enjoying the degree of autonomy appropriate to their task and to the advancement of science and technology.\u201d<sup id=\"rdp-ebb-cite_ref-UNESCORecords17_8-0\" class=\"reference\"><a href=\"#cite_note-UNESCORecords17-8\">[8]<\/a><\/sup>\n<\/p><p>We consider it important, that researchers commit themselves to data management practices like e.g., ISO 31000. However, ISO 31000 defines the risk management process as a feedback loop to be conducted in organizations.<sup id=\"rdp-ebb-cite_ref-ISO31000_4-1\" class=\"reference\"><a href=\"#cite_note-ISO31000-4\">[4]<\/a><\/sup> Projects tend to have a much more limited scope with regard to funding and duration than organizations. Therefore, we regard the ISO 31000 risk management process as too time-consuming and of limited suitability for funded research and similar projects.\n<\/p><p>In this paper, we propose a lightweight approach for the identification of general risks in DMPs. We introduce an initial version of a generic risk catalog for funded research and similar projects. By analyzing a selection of 13 DMPs for projects from multiple disciplines published in the <i>Research Ideas and Outcomes<\/i> (<i>RIO<\/i>) journal<sup id=\"rdp-ebb-cite_ref-CanhosData17_9-0\" class=\"reference\"><a href=\"#cite_note-CanhosData17-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FeyBoulder16_10-0\" class=\"reference\"><a href=\"#cite_note-FeyBoulder16-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FisherAPolit16_11-0\" class=\"reference\"><a href=\"#cite_note-FisherAPolit16-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GattoData17_12-0\" class=\"reference\"><a href=\"#cite_note-GattoData17-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McWhorterCoast16_13-0\" class=\"reference\"><a href=\"#cite_note-McWhorterCoast16-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NeylonData17_14-0\" class=\"reference\"><a href=\"#cite_note-NeylonData17-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NicholsMigrat16_15-0\" class=\"reference\"><a href=\"#cite_note-NicholsMigrat16-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PannellData16_16-0\" class=\"reference\"><a href=\"#cite_note-PannellData16-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TraynorData17_17-0\" class=\"reference\"><a href=\"#cite_note-TraynorData17-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WaelData17_18-0\" class=\"reference\"><a href=\"#cite_note-WaelData17-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WhiteData16_19-0\" class=\"reference\"><a href=\"#cite_note-WhiteData16-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WoolfreyData17_20-0\" class=\"reference\"><a href=\"#cite_note-WoolfreyData17-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-XuAData16_21-0\" class=\"reference\"><a href=\"#cite_note-XuAData16-21\">[21]<\/a><\/sup>, we demonstrate that our approach is applicable and transferable to multiple institutional constellations. As a result, the effort for integrating risk management in data management planning can be reduced.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Related_work\">Related work<\/span><\/h2>\n<p>Jones <i>et al.<\/i> developed a guide for HEIs \u201cto help institutions understand the key aims and issues associated with planning and implementing research data management (RDM) services.\u201d<sup id=\"rdp-ebb-cite_ref-JonesHowTo13_7-1\" class=\"reference\"><a href=\"#cite_note-JonesHowTo13-7\">[7]<\/a><\/sup> In this guide, the authors mention data management risks for HEIs. They note that While the upfront costs for cheap storage of active data \u201cmay be only a fraction of those quoted by central services, the risks of data loss and security breaches are significantly higher, potentially leading to far greater costs in the long term.\u201d<sup id=\"rdp-ebb-cite_ref-JonesHowTo13_7-2\" class=\"reference\"><a href=\"#cite_note-JonesHowTo13-7\">[7]<\/a><\/sup> Additionally, there are \u201cpotential legal risks from using third-party services.\u201d<sup id=\"rdp-ebb-cite_ref-JonesHowTo13_7-3\" class=\"reference\"><a href=\"#cite_note-JonesHowTo13-7\">[7]<\/a><\/sup> However, data selection counters the risks of \u201creputational damage from exposing dirty, confidential, or undocumented data that has been retained long after the researchers who created it have left.\u201d<sup id=\"rdp-ebb-cite_ref-JonesHowTo13_7-4\" class=\"reference\"><a href=\"#cite_note-JonesHowTo13-7\">[7]<\/a><\/sup>\n<\/p><p>The OSCRP working group developed the OSCRP (Open Science Cyber Risk Profile), which \u201cis designed to help principal investigators (PI) and their supporting information technology (IT) professionals assess <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a> risks related to open science projects.\u201d<sup id=\"rdp-ebb-cite_ref-PeisertOpen17_22-0\" class=\"reference\"><a href=\"#cite_note-PeisertOpen17-22\">[22]<\/a><\/sup> The OSCRP working group proposes that principal investigators examine risks, consequences and avenues of attack for each mission critical science asset on an inventory list, whereas assets include devices, systems, data, personnel, workflows, and other kinds of resources.<sup id=\"rdp-ebb-cite_ref-PeisertOpen17_22-1\" class=\"reference\"><a href=\"#cite_note-PeisertOpen17-22\">[22]<\/a><\/sup> We regard this as a very detailed alternative to our approach, but FAIR Guiding Principles<sup id=\"rdp-ebb-cite_ref-WilkinsonTheFAIR16_23-0\" class=\"reference\"><a href=\"#cite_note-WilkinsonTheFAIR16-23\">[23]<\/a><\/sup> and long-term preservation need to be added.\n<\/p><p>In 2014, Ferreira <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-FerreiraData14_24-0\" class=\"reference\"><a href=\"#cite_note-FerreiraData14-24\">[24]<\/a><\/sup> \u201cpropose an analysis process for eScience projects using a data management plan and ISO 31000 in order to create a risk management plan that can complement the data management plan.\u201d The authors describe an analytical process for creating a risk management plan and \u201cpresent the previous process\u2019 validation, based on the MetaGen-FRAME project.\u201d<sup id=\"rdp-ebb-cite_ref-FerreiraData14_24-1\" class=\"reference\"><a href=\"#cite_note-FerreiraData14-24\">[24]<\/a><\/sup> Within this validation Ferreira et al. also identify a project\u2019s task-specific risks, e.g., \u201cR6: Loss of <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a>, denying the representation of the output information to the user via Taverna.\u201d<sup id=\"rdp-ebb-cite_ref-FerreiraData14_24-2\" class=\"reference\"><a href=\"#cite_note-FerreiraData14-24\">[24]<\/a><\/sup> This risk is tailored to the use of Taverna and hence may not be relevant for the majority of funded research and similar projects. There may be projects for which analyzing specific risks for all resources may be crucial. However, a detailed risk analysis may require a considerable amount of work.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>We propose a lightweight approach that can serve as a starting point to include risk management in research data management planning. It doesn\u2019t preclude detailed approaches like OSCRP<sup id=\"rdp-ebb-cite_ref-PeisertOpen17_22-2\" class=\"reference\"><a href=\"#cite_note-PeisertOpen17-22\">[22]<\/a><\/sup> or ISO 31000.<sup id=\"rdp-ebb-cite_ref-ISO31000_4-2\" class=\"reference\"><a href=\"#cite_note-ISO31000-4\">[4]<\/a><\/sup> Instead, we propose an approach which tries to reduce and maybe avoid the burden of a full risk management process like, e.g., ISO 31000. Our approach is based on a pre-tailored and extensible general risk catalog (Table 1) to lessen the effort required for risk management. We derived part of this risk catalog from 29 interviews with researchers from multiple disciplines<sup id=\"rdp-ebb-cite_ref-25\" class=\"reference\"><a href=\"#cite_note-25\">[a]<\/a><\/sup>, which we conducted as part of project SynFo: Creating synergies on the operational level of research data management.<sup id=\"rdp-ebb-cite_ref-SynFo_26-0\" class=\"reference\"><a href=\"#cite_note-SynFo-26\">[25]<\/a><\/sup> One goal of project SynFo was the development of a transferable approach to improve research data management in multiple organizational constellations. In generalized content from the interviews, we identified risks entailed by interfaces of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, e.g., between researchers and data subjects or between researchers and external service providers. For the development of our approach, we also consulted the catalogs for threats and measures from the supplement of the \u201cIT-Grundschutz\u201d catalogs<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-0\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> by the German Federal Office for Information Security (BSI), the FAIR Guiding Principles<sup id=\"rdp-ebb-cite_ref-WilkinsonTheFAIR16_23-1\" class=\"reference\"><a href=\"#cite_note-WilkinsonTheFAIR16-23\">[23]<\/a><\/sup>, and the report and action plan from the European Commission expert group on FAIR data.<sup id=\"rdp-ebb-cite_ref-CollinsTurning18_28-0\" class=\"reference\"><a href=\"#cite_note-CollinsTurning18-28\">[27]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 1.<\/b> General risk catalog\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Risk category\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Risk [CODE]\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Possible risk source\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\"><b>Legal<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Penalty for conducting unreported notifiable practices [RLEGU]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Physical sample collection\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Penalty for unpermitted usage of external data [RLEGE]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Processing external data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Penalty for unpermitted usage of personal data [RLEGP]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Processing personal data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Penalty for conducting inadequate data protection practices [RLEGD]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Using an external service provider for processing personal data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\"><b>Privacy<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Loss of confidentiality through sending data to an unintended recipient [RPRIR]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Correspondence\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Loss of confidentiality through interception or eavesdropping of information [RPRII]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Online data transmission\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Loss of confidentiality through loss or theft of portable storage media or devices [RPRIS]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Portable storage media or devices\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Loss of confidentiality through careless data handling by an external party [RPRIE]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sharing data with an external party without publication purposes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\"><b>Technical<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Unavailability through data corruption [RTECC]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data processing\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Unavailability through data loss [RTECL]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data storage\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\"><b>Science<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Poor knowledge discovery or reusability for stakeholders cannot find the data [RSCIF]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Searchable information not planned\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Poor knowledge discovery or reusability for stakeholders cannot access the data [RSCIA]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sharing location not planned\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Poor knowledge discovery or reusability for stakeholders cannot integrate the data [RSCII]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">File format not planned\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Poor knowledge discovery or reusability for stakeholders cannot reuse the data [RSCIR]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Licensing and context information not planned\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Preservation<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Unsustainability in the long-term through unavailability or discontinuity of financial support [RPREU]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Preservation location not planned\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Our risk identification includes risks, their possible risk sources, mitigation approaches, and consequences. By analyzing occurrences and mitigations of risks from our catalog within a selection of 13 DMPs from multiple disciplines<sup id=\"rdp-ebb-cite_ref-29\" class=\"reference\"><a href=\"#cite_note-29\">[b]<\/a><\/sup>, published in the \u2018\u2019RIO\u2019\u2019 journal, we demonstrate that our lightweight approach is applicable to DMPs and transferable to multiple institutional constellations. We evaluate the occurrences of the 15 risks in our catalog by identifying possible risk sources in each of the selected DMPs and analyze the risk mitigations in accordance to what the authors wrote.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Risks\">Risks<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Legal_risks\">Legal risks<\/span><\/h3>\n<p>A breach of a regulation like the <a href=\"https:\/\/www.limswiki.org\/index.php\/General_Data_Protection_Regulation\" title=\"General Data Protection Regulation\" class=\"wiki-link\" data-key=\"3f4bdf6f0dcb360b1e79aad8674c2447\">General Data Protection Regulation<\/a> (GDPR) or the Nagoya Protocol can result in high fines. At worst, compliance breaches can lead to reputational damages, legal disputes, and enormous cost.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Penalty_for_conducting_unreported_notifiable_practices_.5BRLEGU.5D\">Penalty for conducting unreported notifiable practices [RLEGU]<\/span><\/h4>\n<p>Research may include reportable research practices like the collection of physical samples regulated by the <i>Nagoya Protocol on Access to Genetic Resources and the Fair and Equitable Sharing of Benefits Arising from their Utilization<\/i>, which was transposed into E.U. law by Regulation (EU) No 511\/2014. Under this regulation, there is a reporting obligation if the research on genetic resources is financially supported (Art. 7, Sec. 1) and if the research is in the final stage of development of a product that is based on the utilization of genetic resources (Art. 7, Sec.2).<sup id=\"rdp-ebb-cite_ref-ELEXReg511_30-0\" class=\"reference\"><a href=\"#cite_note-ELEXReg511-30\">[28]<\/a><\/sup> Article 11 says that \u201cMember States shall lay down the rules on penalties applicable to infringements of Articles 4 and 7 and shall take all the measures necessary to ensure that they are applied.\u201d<sup id=\"rdp-ebb-cite_ref-ELEXReg511_30-1\" class=\"reference\"><a href=\"#cite_note-ELEXReg511-30\">[28]<\/a><\/sup> The Nagoya Protocol \u201cand EU documents themselves give no guidance on penalties, each country has the liberty to determine these.\u201d<sup id=\"rdp-ebb-cite_ref-vanVegchelImplem18_31-0\" class=\"reference\"><a href=\"#cite_note-vanVegchelImplem18-31\">[29]<\/a><\/sup> Consequences may be fines of up to EUR 810,000 or even imprisonment.<sup id=\"rdp-ebb-cite_ref-vanVegchelImplem18_31-1\" class=\"reference\"><a href=\"#cite_note-vanVegchelImplem18-31\">[29]<\/a><\/sup> To avoid penalties, the parties should comply strictly with the rules. The Convention on Biological Diversity publishes a detailed list of parties to the Nagoya Protocol.<sup id=\"rdp-ebb-cite_ref-32\" class=\"reference\"><a href=\"#cite_note-32\">[c]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Penalty_for_unpermitted_usage_of_external_data_.5BRLEGE.5D\">Penalty for unpermitted usage of external data [RLEGE]<\/span><\/h4>\n<p>In many countries, data by themselves do not have inherent legal protection. License contracts can reach various agreements concerning terms of use. Free licenses make (data) objects available for utilization to everyone, but usage can be restricted or conditioned. Creative Commons (CC) licenses and the GNU General Public License (GPL), which is specialized for free software, are widely used. Nonetheless, using CC licenses can lead to conflicting rights of third parties. Publicity, personality, and privacy rights \u201cnot held by the licensor are not affected and may still affect your desired use of a licensed work.\u201d<sup id=\"rdp-ebb-cite_ref-CCFaq_33-0\" class=\"reference\"><a href=\"#cite_note-CCFaq-33\">[30]<\/a><\/sup> \u201cIf there are any third parties who may have publicity, privacy, or personality rights that apply, those rights are not affected by your application of a CC license, and a reuser must seek permission for relevant uses.\u201d<sup id=\"rdp-ebb-cite_ref-CCFaq_33-1\" class=\"reference\"><a href=\"#cite_note-CCFaq-33\">[30]<\/a><\/sup> This example holds for pictures of persons. Also, the GNU GPL license imposes transitive obligations, e.g., \u201cderivative programs must also be subject to the same initial GPL conditions of ability to copy, modify, or redistribute.\u201d<sup id=\"rdp-ebb-cite_ref-LipinskiLib12_34-0\" class=\"reference\"><a href=\"#cite_note-LipinskiLib12-34\">[31]<\/a><\/sup> To mitigate the risk of unpermitted usage of external data, it is recommended to abide by the license terms. In general, an overview about the data and the related licenses can be developed in the DMP or within the framework of a data policy.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Penalty_for_unpermitted_usage_of_personal_data_.5BRLEGP.5D\">Penalty for unpermitted usage of personal data [RLEGP]<\/span><\/h4>\n<p>In the E.U., the GDPR governs the processing of personal data. Articles 6 and 7 of the GDPR regulate the lawfulness of processing and the conditions of consent. On an international level, the European Commission can conduct an assessment to \u201censure that the level of data protection in a third country or international organization is essentially equivalent to the one established by the E.U. legislation.\u201d<sup id=\"rdp-ebb-cite_ref-AP29Adequ18_35-0\" class=\"reference\"><a href=\"#cite_note-AP29Adequ18-35\">[32]<\/a><\/sup> Canada (commercial organizations), Israel, Switzerland, Japan, and the U.S. (limited to the Privacy Shield Framework) offer an adequate level of data protection.<sup id=\"rdp-ebb-cite_ref-ECAdeq19_36-0\" class=\"reference\"><a href=\"#cite_note-ECAdeq19-36\">[33]<\/a><\/sup> To avoid penalties, it is recommended to receive written consent forms from data subjects, including information about the purpose and procedures of data processing.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Penalty_for_conducting_inadequate_data_protection_practices_.5BRLEGD.5D\">Penalty for conducting inadequate data protection practices [RLEGD]<\/span><\/h4>\n<p>Article 5 of the GDPR enumerates principles related to processing of personal data: the principle of lawfulness, fairness and transparency, purpose limitation, data minimization, accuracy, storage limitation, integrity, and confidentiality, as well as accountability. According to Article 45 of the GDPR, \u201cA transfer of personal data to a third country or an international organization may take place where the Commission has decided that the third country, a territory or one or more specified sectors within that third country, or the international organization in question ensures an adequate level of protection. Such a transfer shall not require any specific authorization.\u201d<sup id=\"rdp-ebb-cite_ref-ELEXReg679_37-0\" class=\"reference\"><a href=\"#cite_note-ELEXReg679-37\">[34]<\/a><\/sup> Countries without adequacy, which are not classified as safe third countries, can guarantee protection in other ways, for example by using appropriate safeguards (Art. 6, GDPR) or binding corporate rules (Art. 7, GDPR). To avoid penalties, it is recommended to abide by the applicable laws. In case of doubt, researchers can contact the appropriate (data protection) authorities.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Privacy_risks\">Privacy risks<\/span><\/h3>\n<p>A loss of confidentiality can have adverse effects on an organization like financial effects.<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-1\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> These effects may also apply to a researcher who additionally may want to keep research data confidential before scientific output is published, so that research data will not be subject to theft of work.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Loss_of_confidentiality_through_sending_data_to_an_unintended_recipient_.5BRPRIR.5D\">Loss of confidentiality through sending data to an unintended recipient [RPRIR]<\/span><\/h4>\n<p>Correspondence has the intrinsic potential that a researcher transmits data to an unintended recipient. This may happen accidentally or as the result of a fraudulent attack like social engineering and leads to loss of confidentiality. \u201cSocial engineering is a method used to gain unauthorized access to information or IT systems by social action.\u201d<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-2\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> Researchers should take extra care when sending confidential information and be aware of fraudulent attacks.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Loss_of_confidentiality_through_interception_or_eavesdropping_of_information_.5BRPRII.5D\">Loss of confidentiality through interception or eavesdropping of information [RPRII]<\/span><\/h4>\n<p>In the supplement of the IT-Grundschutz catalogs, the BSI specifies the threats of interception or eavesdropping of information, which entail the risk of loss of confidentiality.<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-3\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> \u201cSince data is sent using unforeseeable routes and nodes on the internet, the sent data should only be transmitted in an <a href=\"https:\/\/www.limswiki.org\/index.php\/Encryption\" title=\"Encryption\" class=\"wiki-link\" data-key=\"86a503652ed5cc9d8e2b0252a480b5e1\">encrypted<\/a> form, as far as possible.\u201d<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-4\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Loss_of_confidentiality_through_loss_or_theft_of_portable_storage_media_or_devices_.5BRPRIS.5D\">Loss of confidentiality through loss or theft of portable storage media or devices [RPRIS]<\/span><\/h4>\n<p>\u201cPortable terminal devices and mobile data media in particular can be lost easily\u201d<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-5\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> or even be stolen. \u201cWhenever possible, mobile data media such as USB sticks and laptops should always be encrypted completely even if they are only occasionally used for confidential information.\u201d<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-6\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Loss_of_confidentiality_through_careless_data_handling_by_an_external_party_.5BRPRIE.5D\">Loss of confidentiality through careless data handling by an external party [RPRIE]<\/span><\/h4>\n<p>We regard the event that researchers share data with an external party without the purpose of publication as entailing the risk of loss of confidentiality. The external party may handle confidential data carelessly. \u201cIt can frequently be observed that there are a number of organizational or technical security procedures available in organizations, but they are then undermined through careless handling of the specifications and the technology.\u201d<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-7\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> We recommend that researchers who share their research data to always grant specific usage rights in written form to the external party or to check if appropriate security measures are applied by the external party.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Technical_risks\">Technical risks<\/span><\/h3>\n<p>Data can lose their <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">integrity<\/a> or be lost<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-8\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup>, leading to the major risk of unavailability of data. Unavailability of the correct data through silent corruption can lead to usage of incorrect data and hence to the production of incorrect results. If data are unavailable, either the project may fail or researchers need to repeat their data collection and the project will be behind schedule.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Unavailability_through_data_corruption_.5BRTECC.5D\">Unavailability through data corruption [RTECC]<\/span><\/h4>\n<p>\u201cThe integrity of information may be impaired due to different causes, e.g., manipulations, errors caused by people, incorrect use of applications, malfunctions of software, or transmission errors.\u201d<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-9\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> \u201cIf only accidental changes need to be detected, then checksum procedures (e.g., cyclic redundancy checks) or error-correcting codes can be used.\u201d<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-10\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> Nonetheless, there may be other scenarios where these verification techniques are insufficient.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Unavailability_through_data_loss_.5BRTECL.5D\">Unavailability through data loss [RTECL]<\/span><\/h4>\n<p>Data may \u201cbe lost when devices or data media are damaged, lost, or stolen,\u201d<sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-11\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> hence becoming unavailable. Approaches to mitigate irretrievable losses of data are for example regular <a href=\"https:\/\/www.limswiki.org\/index.php\/Backup\" title=\"Backup\" class=\"wiki-link\" data-key=\"e12548e6bf5f28bfee99099fe8662dde\">backups<\/a><sup id=\"rdp-ebb-cite_ref-BSIITGrund15_27-12\" class=\"reference\"><a href=\"#cite_note-BSIITGrund15-27\">[26]<\/a><\/sup> or keeping copies in multiple storage locations.<sup id=\"rdp-ebb-cite_ref-ReichLOCKSS09_38-0\" class=\"reference\"><a href=\"#cite_note-ReichLOCKSS09-38\">[35]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Science_risks\">Science risks<\/span><\/h3>\n<p>Consequences of poor discoverability and reusability of data are that researchers may unnecessarily repeat work and that scientific outputs derived from it may fail to be comprehensible, reproducible, or traceable. Problems with reproducibility and replication \u201ccan cause permanent damage to the credibility of science,\u201d<sup id=\"rdp-ebb-cite_ref-PengTheRepro15_39-0\" class=\"reference\"><a href=\"#cite_note-PengTheRepro15-39\">[36]<\/a><\/sup> and thus why this category is referred to as \"science risks.\"\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Poor_knowledge_discovery_or_reusability_for_stakeholders_cannot_find_.5BRSCIF.5D.2C_access_.5BRSCIA.5D.2C_integrate_.5BRSCII.5D.2C_or_reuse_.5BRSCIR.5D_the_data\">Poor knowledge discovery or reusability for stakeholders cannot find [RSCIF], access [RSCIA], integrate [RSCII], or reuse [RSCIR] the data<\/span><\/h4>\n<p>Making data fundable, accessible, interoperable, and reusable (FAIR) to human and computational stakeholders is a best practice approach described in the FAIR Guiding Principles of Wilkinson <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-WilkinsonTheFAIR16_23-2\" class=\"reference\"><a href=\"#cite_note-WilkinsonTheFAIR16-23\">[23]<\/a><\/sup> Therefore, we include the risks that stakeholders cannot find, access, process, or reuse data in our risk catalog. Authors of DMPs can mitigate these risks as described by Wilkinson <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-WilkinsonTheFAIR16_23-3\" class=\"reference\"><a href=\"#cite_note-WilkinsonTheFAIR16-23\">[23]<\/a><\/sup> We abbreviated the risk names under this risk category using the term \u201cpoor knowledge discovery or reusability,\u201d but refer to all the FAIR Guiding Principles for more information.<sup id=\"rdp-ebb-cite_ref-WilkinsonTheFAIR16_23-4\" class=\"reference\"><a href=\"#cite_note-WilkinsonTheFAIR16-23\">[23]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Preservation_risk\">Preservation risk<\/span><\/h3>\n<p>If data are not suitably preserved, scientific outputs derived from them may fail to be comprehensible, reproducible, or traceable in the long run. Data should be stored in a trusted and sustainable digital repository.<sup id=\"rdp-ebb-cite_ref-CollinsTurning18_28-1\" class=\"reference\"><a href=\"#cite_note-CollinsTurning18-28\">[27]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Unsustainability_in_the_long-term_through_unavailability_or_discontinuity_of_financial_support_.5BRPREU.5D\">Unsustainability in the long-term through unavailability or discontinuity of financial support [RPREU]<\/span><\/h4>\n<p>A digital preservation location has the intrinsic technical risk that data become unavailable through data loss or corruption. However, preservation locations also entail the risk of becoming unavailable when their funding ends. For example, Canhos states that discontinuity of financial support is a threat to Brazil\u2019s Virtual Herbarium and its data sources.<sup id=\"rdp-ebb-cite_ref-CanhosData17_9-1\" class=\"reference\"><a href=\"#cite_note-CanhosData17-9\">[9]<\/a><\/sup> Authors of DMPs should consider these risks when selecting a preservation location. They can mitigate the risk that data are not preserved long-term by reviewing the external preservation location\u2019s longevity, certificates, and funding. We also suggest that attention is paid to possible migration and exit strategies like exporting and handing over data to a national data archive. This may particularly be important when the preservation location is not external.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Evaluation\">Evaluation<\/span><\/h2>\n<p>When applying the risk catalog (Table 1) to the sample of 13 DMPs, we distinguish between risk occurrences themselves and risk occurrences with at least one mitigation, as show in Table 2.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"16\"><b>Table 2.<\/b> Risk occurrences (+) and risk occurrences with at least one mitigation (-) in the sample of 13 DMPs\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">DMP\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RLEGU\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RLEGE\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RLEGP\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RLEGD\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RPRIR\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RPRII\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RPRIS\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RPRIE\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RTECC\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RTECL\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RSCIF\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RSCIA\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RSCII\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RSCIR\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">RPREU\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Canhos, 2017<sup id=\"rdp-ebb-cite_ref-CanhosData17_9-2\" class=\"reference\"><a href=\"#cite_note-CanhosData17-9\">[9]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fey and Anderson, 2016<sup id=\"rdp-ebb-cite_ref-FeyBoulder16_10-1\" class=\"reference\"><a href=\"#cite_note-FeyBoulder16-10\">[10]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fisher and Nading, 2016<sup id=\"rdp-ebb-cite_ref-FisherAPolit16_11-1\" class=\"reference\"><a href=\"#cite_note-FisherAPolit16-11\">[11]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Gatto, 2017<sup id=\"rdp-ebb-cite_ref-GattoData17_12-1\" class=\"reference\"><a href=\"#cite_note-GattoData17-12\">[12]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">McWhorter <i>et al.<\/i>, 2016<sup id=\"rdp-ebb-cite_ref-McWhorterCoast16_13-1\" class=\"reference\"><a href=\"#cite_note-McWhorterCoast16-13\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neylon, 2017<sup id=\"rdp-ebb-cite_ref-NeylonData17_14-1\" class=\"reference\"><a href=\"#cite_note-NeylonData17-14\">[14]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nichols and Stolze, 2016<sup id=\"rdp-ebb-cite_ref-NicholsMigrat16_15-1\" class=\"reference\"><a href=\"#cite_note-NicholsMigrat16-15\">[15]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pannell, 2016<sup id=\"rdp-ebb-cite_ref-PannellData16_16-1\" class=\"reference\"><a href=\"#cite_note-PannellData16-16\">[16]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Traynor, 2017<sup id=\"rdp-ebb-cite_ref-TraynorData17_17-1\" class=\"reference\"><a href=\"#cite_note-TraynorData17-17\">[17]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wael, 2017<sup id=\"rdp-ebb-cite_ref-WaelData17_18-1\" class=\"reference\"><a href=\"#cite_note-WaelData17-18\">[18]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">White, 2016<sup id=\"rdp-ebb-cite_ref-WhiteData16_19-1\" class=\"reference\"><a href=\"#cite_note-WhiteData16-19\">[19]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Woolfrey, 2017<sup id=\"rdp-ebb-cite_ref-WoolfreyData17_20-1\" class=\"reference\"><a href=\"#cite_note-WoolfreyData17-20\">[20]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Xu <i>et al.<\/i>, 2016<sup id=\"rdp-ebb-cite_ref-XuAData16_21-1\" class=\"reference\"><a href=\"#cite_note-XuAData16-21\">[21]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Because risk sources and mitigations were not always explicitly mentioned in the 13 sample DMPs, we needed to make interpretations. Appendix A shows our interpretation notes. According to these interpretations, we then found the mitigations, as shown in Appendix B.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Evaluation_results\">Evaluation results<\/span><\/h2>\n<p>Each of the 15 risks of our catalog occurred in at least two of the selected 13 DMPs. Table 3 summarizes our evaluation results.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 3.<\/b> Summary of risk evaluation results\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Risk from catalog\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">% of risk occurrences mitigated\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Most often used mitigation strategy (in # of DMPs)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Unavailability through data loss [RTECL]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Backup (8)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Poor knowledge discovery or reusability for stakeholders cannot access the data [RSCIA]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Specific repository (7)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Poor knowledge discovery or reusability for stakeholders cannot reuse the data [RSCIR]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">92.31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Specific license (9)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Unsustainability in the long-term through unavailability or discontinuity of financial support [RPREU]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">76.92\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Specific file formats (4), Specific data archive (4)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Poor knowledge discovery or reusability for stakeholders cannot integrate the data [RSCII]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">69.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Specific file formats (8)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Poor knowledge discovery or reusability for stakeholders cannot find the data [RSCIF]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">61.54\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Metadata (2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Loss of confidentiality through careless data handling by an external party [RPRIE]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">40.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Agreement for IP rights (1), Secure external infrastructure (1)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Penalty for unpermitted usage of external data [RLEGE]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">37.50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Respect usage permissions of external data (2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Penalty for unpermitted usage of personal data [RLEGP]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Signed consent forms (1)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Unavailability through data corruption [RTECC]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15.38\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Compare data from before and after transmission (1), Data <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> (1)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Loss of confidentiality through interception or eavesdropping of information [RPRII]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Penalty for conducting inadequate data protection practices [RLEGD]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Loss of confidentiality through sending data to an unintended recipient [RPRIR]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Loss of confidentiality through loss or theft of portable storage media or devices [RPRIS]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Penalty for conducting unreported notifiable practices [RLEGU]\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">00.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Within the small sample of 13 DMPs, we found 34 distinct strategies to mitigate 10 of the 15 risks of our proposed catalog. Hence, we also found that for five of the 15 risks from our catalog the authors did not describe any mitigation in the corresponding DMP. These risks are legal and privacy risks and they do have possible consequences like loss of reputation or project failure through theft of work. The authors of the selected DMPs overall attach highest importance to mitigating data unavailability through data loss; making data findable, accessible, interoperable, and reusable; and ensuring their long-term digital preservation. We found that all the authors of the 13 DMPs managed to mitigate two risks from our catalog: unavailability through data loss (RTECL) and poor knowledge discoverability or reusability for stakeholders cannot access the data (RSCIA).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>Since we identified each risk of our catalog in at least two of the selected DMPs, we conclude that our risk catalog is applicable to DMPs from multiple areas of research. In the selected DMPs, we overall find 53 of 125 (42.4%) risk occurrences not mitigated and hence see the necessity of DMP quality improvement through risk identification and mitigation planning in the data management planning phase.\n<\/p><p>We consider our approach useful for identifying general risks in DMPs. We propose that after filling out a funder\u2019s DMP template, authors of DMPs refer to a risk catalog to identify possible risk sources and hence risks. Next, the authors should add mitigations to their DMP in the corresponding paragraph, if their DMP does not already contain one. For example, in a DMP\u2019s paragraph in which authors write about the usage of external hard disks, they should add a sentence indicating that these external hard disks will be encrypted to mitigate the risk of loss of confidentiality through loss or theft of storage media, if their DMP does not yet contain any measures mitigating this risk.\n<\/p><p>The risk catalog may also be useful to funders, since it makes it possible for them to evaluate basic investment risks of proposed projects.\n<\/p><p>Note that many of the legal assertions in this article hold within the E.U. Applicability to non-E.U. countries may vary.\n<\/p><p>We think further research on suitable risk management approaches concerning the data management of funded research and similar projects needs to be conducted.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Appendices\">Appendices<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Appendix_A:_Interpretation_notes\">Appendix A: Interpretation notes<\/span><\/h3>\n<p>Canhos states that discontinuity of financial support is a threat to Brazil\u2019s Virtual Herbarium and its data sources.<sup id=\"rdp-ebb-cite_ref-CanhosData17_9-3\" class=\"reference\"><a href=\"#cite_note-CanhosData17-9\">[9]<\/a><\/sup> We interpret this as the risk that data are not preserved long-term.\n<\/p><p>In Fey and Anderson's DMP<sup id=\"rdp-ebb-cite_ref-FeyBoulder16_10-2\" class=\"reference\"><a href=\"#cite_note-FeyBoulder16-10\">[10]<\/a><\/sup>, we interpret the choice of .txt and .csv formats as open file formats for interoperability, and we interpret the use of metadata standards as to mitigate the risk that data are not findable.\n<\/p><p>In Fisher and Nading\u2019s DMP <sup id=\"rdp-ebb-cite_ref-FisherAPolit16_11-2\" class=\"reference\"><a href=\"#cite_note-FisherAPolit16-11\">[11]<\/a><\/sup>, we find only geospatial metadata mentioned. Neither documentation nor license information are mentioned in Fisher and Nading\u2019s DMP. <sup id=\"rdp-ebb-cite_ref-FisherAPolit16_11-3\" class=\"reference\"><a href=\"#cite_note-FisherAPolit16-11\">[11]<\/a><\/sup>\n<\/p><p>In Gatto\u2019s DMP<sup id=\"rdp-ebb-cite_ref-GattoData17_12-2\" class=\"reference\"><a href=\"#cite_note-GattoData17-12\">[12]<\/a><\/sup>, we assume data are enriched or combined so that a license of the resulting data set should be derived from the source data licenses in Gatto\u2019s DMP. We evaluate Gatto\u2019s DMP according to the FAIR Guiding Principles for research software, as proposed by Jim\u00e9nez <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Jim.C3.A9nezFour17_40-0\" class=\"reference\"><a href=\"#cite_note-Jim.C3.A9nezFour17-40\">[37]<\/a><\/sup>\n<\/p><p>Concerning the DMP of McWhorter <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-McWhorterCoast16_13-2\" class=\"reference\"><a href=\"#cite_note-McWhorterCoast16-13\">[13]<\/a><\/sup>, we assume that data do not have protection requirements and no risk of interception because data are made freely available for public use.\n<\/p><p>Neylon\u2019s DMP<sup id=\"rdp-ebb-cite_ref-NeylonData17_14-2\" class=\"reference\"><a href=\"#cite_note-NeylonData17-14\">[14]<\/a><\/sup> does not include collecting signed forms of consent from interviewees. Neylon also decides not to make all data anonymous and accessible.<sup id=\"rdp-ebb-cite_ref-NeylonData17_14-3\" class=\"reference\"><a href=\"#cite_note-NeylonData17-14\">[14]<\/a><\/sup> Finally, the DMP does not include using file formats that are interoperable or allow re-use.\n<\/p><p>Nichols and Stolze\u2019s DMP<sup id=\"rdp-ebb-cite_ref-NicholsMigrat16_15-2\" class=\"reference\"><a href=\"#cite_note-NicholsMigrat16-15\">[15]<\/a><\/sup> describes migration of data from old storage media to an .xlsx format and their subsequent publication.\n<\/p><p>Concerning Pannell\u2019s DMP<sup id=\"rdp-ebb-cite_ref-PannellData16_16-2\" class=\"reference\"><a href=\"#cite_note-PannellData16-16\">[16]<\/a><\/sup>, we think that it would be adequate to inform the responsible authority about the planned research project. Pannell does not address rights of use of external data. We interpret Pannell's term \"filterable,\"<sup id=\"rdp-ebb-cite_ref-PannellData16_16-3\" class=\"reference\"><a href=\"#cite_note-PannellData16-16\">[16]<\/a><\/sup> in the context of metadata documentation, as meaning \"findable.\"\n<\/p><p>In Traynor\u2019s DMP<sup id=\"rdp-ebb-cite_ref-TraynorData17_17-2\" class=\"reference\"><a href=\"#cite_note-TraynorData17-17\">[17]<\/a><\/sup>, it is not clear if personal data are anonymized before they are uploaded in the infrastructure of an external service provider. Traynor\u2019s DMP also contains no decisions for metadata capture or a specific long-term preservation location.<sup id=\"rdp-ebb-cite_ref-TraynorData17_17-3\" class=\"reference\"><a href=\"#cite_note-TraynorData17-17\">[17]<\/a><\/sup>\n<\/p><p>As for Wael\u2019s DMP, they plan to hire a consultant to do technical planning and system set up.<sup id=\"rdp-ebb-cite_ref-WaelData17_18-2\" class=\"reference\"><a href=\"#cite_note-WaelData17-18\">[18]<\/a><\/sup> The DMP does not include making data interoperable. We think that utilizing academic contacts to inform the research community that the data exists, as stated by Wael<sup id=\"rdp-ebb-cite_ref-WaelData17_18-3\" class=\"reference\"><a href=\"#cite_note-WaelData17-18\">[18]<\/a><\/sup>, is not the same as making data findable.\n<\/p><p>According to White\u2019s DMP, the project members will develop data and software \u201cin the open,\u201d<sup id=\"rdp-ebb-cite_ref-WhiteData16_19-2\" class=\"reference\"><a href=\"#cite_note-WhiteData16-19\">[19]<\/a><\/sup> which we interpret as making data accessible. In his DMP, White does not mention long-term preservation.<sup id=\"rdp-ebb-cite_ref-WhiteData16_19-3\" class=\"reference\"><a href=\"#cite_note-WhiteData16-19\">[19]<\/a><\/sup> We regard the metadata capture and user-focused documentation stated in White\u2019s DMP<sup id=\"rdp-ebb-cite_ref-WhiteData16_19-4\" class=\"reference\"><a href=\"#cite_note-WhiteData16-19\">[19]<\/a><\/sup> as making data reusable.\n<\/p><p>In Woolfrey\u2019s DMP<sup id=\"rdp-ebb-cite_ref-WoolfreyData17_20-2\" class=\"reference\"><a href=\"#cite_note-WoolfreyData17-20\">[20]<\/a><\/sup>, metadata are captured for re-usability. Woolfrey\u2019s DMP does not include making data findable.\n<\/p><p>Xu <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-XuAData16_21-2\" class=\"reference\"><a href=\"#cite_note-XuAData16-21\">[21]<\/a><\/sup> do not explicitly state in which states or countries they plan to collect physical samples. We make the interpretation that physical samples are registered with a persistent identifier, as described by Xu <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-XuAData16_21-3\" class=\"reference\"><a href=\"#cite_note-XuAData16-21\">[21]<\/a><\/sup>, to make their metadata findable. Xu <i>et al.<\/i> write that for re-use and distribution, IEDA (Interdisciplinary Earth Data Alliance) would have a persistent identifier assigned to the data sets.<sup id=\"rdp-ebb-cite_ref-XuAData16_21-4\" class=\"reference\"><a href=\"#cite_note-XuAData16-21\">[21]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Appendix_B:_Risk_mitigations\">Appendix B: Risk mitigations<\/span><\/h3>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 4.<\/b> Risk mitigations\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">DMP\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Risk mitigations\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Canhos, 2017<sup id=\"rdp-ebb-cite_ref-CanhosData17_9-4\" class=\"reference\"><a href=\"#cite_note-CanhosData17-9\">[9]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Aligned licensing of all data (RLEGE), Backup (RTECL), Maintain blog and social media account (RSCIF), Publicly accessible server (RSCIA), Specific fle formats (RSCIR), Specific license (RSCIR), Metadata or citation of external data (RSCIR), Standard data model (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fey and Anderson, 2016<sup id=\"rdp-ebb-cite_ref-FeyBoulder16_10-3\" class=\"reference\"><a href=\"#cite_note-FeyBoulder16-10\">[10]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Backup (RTECL), Provide rights of use (RSCIR); Specific data archive (RPREU, RSCIA), Metadata (RSCIF), Metadata standard (RSCIF), Publicly accessible server (RSCIA), Specific file formats (RSCII)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fisher and Nading, 2016<sup id=\"rdp-ebb-cite_ref-FisherAPolit16_11-4\" class=\"reference\"><a href=\"#cite_note-FisherAPolit16-11\">[11]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Backup (RTECL), Specific data archive (RPREU, RSCIA), Listing in national discipline-specific wiki (RSCIF), Listing on funders website (RSCIF), Specific file formats (RSCII)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Gatto, 2017<sup id=\"rdp-ebb-cite_ref-GattoData17_12-3\" class=\"reference\"><a href=\"#cite_note-GattoData17-12\">[12]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple storage locations (RTECL), Specific license (RSCIR), Collaborative software development (RPREU, RSCIF), Specific repository (RSCIA), Documentation (RSCIR), Metadata or citation of external data (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">McWhorter <i>et al.<\/i>, 2016<sup id=\"rdp-ebb-cite_ref-McWhorterCoast16_13-3\" class=\"reference\"><a href=\"#cite_note-McWhorterCoast16-13\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data are freely available for public use (RSCIR), Data quality control (RTECC), Backup (RTECL), Multiple preservation locations (RTECL, RPREU), Specific data archive (RPREU), Publicly accessible server (RSCIA), Specific file formats (RSCII), Metadata (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neylon, 2017<sup id=\"rdp-ebb-cite_ref-NeylonData17_14-4\" class=\"reference\"><a href=\"#cite_note-NeylonData17-14\">[14]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple storage locations (RTECL), Specific license (RSCIR), Multiple preservation locations (RPREU), Specific file formats (RPREU, RSCII), Persistent identifier (RSCIF), Specific repository (RSCIA), Documentation (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nichols and Stolze, 2016<sup id=\"rdp-ebb-cite_ref-NicholsMigrat16_15-3\" class=\"reference\"><a href=\"#cite_note-NicholsMigrat16-15\">[15]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Compare data from before and after transmission (RTECC), Backup (RTECL), Multiple storage locations (RTECL), Specific license (RSCIR), Repository guarantees long-term availability (RPREU), Publish data descriptor in open-access journal (RSCIF, RSCIR), Specific repository (RSCIA), Specific file formats (RSCII), Documentation (RSCIR), Metadata (RSCIR), Persistent identifier (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pannell, 2016<sup id=\"rdp-ebb-cite_ref-PannellData16_16-4\" class=\"reference\"><a href=\"#cite_note-PannellData16-16\">[16]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Secure external service infrastructure (RPRIE), Replicas in external service infrastructure (RTECL), Specific license (RSCIR), Preservation at institution\u2019s library (RPREU), Specific file formats (RPREU, RSCII), Specific repository (RSCIA), Metadata (RSCIF), Persistent identifier (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Traynor, 2017<sup id=\"rdp-ebb-cite_ref-TraynorData17_17-4\" class=\"reference\"><a href=\"#cite_note-TraynorData17-17\">[17]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Signed consent forms (RLEGP), Multiple storage locations (RTECL), Backup (RTECL), Specific license (RSCIR), Specific file formats (RPREU), Specific repository (RSCIA), Specific file formats (RSCII), Documentation (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wael, 2017<sup id=\"rdp-ebb-cite_ref-WaelData17_18-4\" class=\"reference\"><a href=\"#cite_note-WaelData17-18\">[18]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple storage locations (RTECL), Agreement for IP rights (RPRIE), Specific license (RSCIR), Publicly accessible server (RSCIA), Anonymization (RSCIR), Documentation (RSCIR), Specific file formats (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">White, 2016<sup id=\"rdp-ebb-cite_ref-WhiteData16_19-5\" class=\"reference\"><a href=\"#cite_note-WhiteData16-19\">[19]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Respect usage permissions of external data (RLEGE), Backup (RTECL), Specific license (RSCIR), Specific repository (RSCIA), Metadata (RSCIR), Documentation (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Woolfrey, 2017<sup id=\"rdp-ebb-cite_ref-WoolfreyData17_20-3\" class=\"reference\"><a href=\"#cite_note-WoolfreyData17-20\">[20]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Respect usage permissions of external data (RLEGE), Backup (RTECL), Data are freely available for public use (RSCIR), Specific data archive (RPREU, RSCIA), Specific file formats (RSCII), Documentation (RSCIR), Metadata (RSCIR), Metadata standard (RSCIR)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Xu <i>et al.<\/i>, 2016<sup id=\"rdp-ebb-cite_ref-XuAData16_21-5\" class=\"reference\"><a href=\"#cite_note-XuAData16-21\">[21]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple storage locations (RTECL), Specific license (RSCIR), Specific file formats (RPREU), Specific repository (RPREU, RSCIA), Persistent identifier for physical samples (RSCIF), Documentation (RSCIR), Metadata (RSCIR); Persistent identifier (RSCIR), Standardized vocabulary (RSCIR)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Footnotes\">Footnotes<\/span><\/h2>\n<div class=\"reflist\" style=\"list-style-type: lower-alpha;\">\n<ol class=\"references\">\n<li id=\"cite_note-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-25\">\u2191<\/a><\/span> <span class=\"reference-text\">Geo sciences (12), biology (5), humanities (5), social and behavioral sciences (4), computer science, systems engineering and electrical engineering (2), and medicine (1)<\/span>\n<\/li>\n<li id=\"cite_note-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-29\">\u2191<\/a><\/span> <span class=\"reference-text\">Biology (4), geo sciences (4), social and behavioral sciences (3), computer science, systems engineering and electrical engineering (1), and humanities (1)<\/span>\n<\/li>\n<li id=\"cite_note-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-32\">\u2191<\/a><\/span> <span class=\"reference-text\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cbd.int\/abs\/nagoya-protocol\/signatories\/\" target=\"_blank\">Parties to the Nagoya Protocol<\/a><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We thank Peter Wullinger and Klaus Stein for their constructive feedback.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work was funded by the German Federal Ministry of Education and Research (BMBF) under the funding ID 16FDM024.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-MichenerTen15-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MichenerTen15_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Michener, W.K. (2015). \"Ten Simple Rules for Creating a Good Data Management Plan\". <i>PLoS Computational Biology<\/i> <b>11<\/b> (10): e1004525. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1004525\" target=\"_blank\">10.1371\/journal.pcbi.1004525<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ten+Simple+Rules+for+Creating+a+Good+Data+Management+Plan&rft.jtitle=PLoS+Computational+Biology&rft.aulast=Michener%2C+W.K.&rft.au=Michener%2C+W.K.&rft.date=2015&rft.volume=11&rft.issue=10&rft.pages=e1004525&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1004525&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DonnellyData12-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DonnellyData12_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Donnelly, M. (2012). \"Chapter 5: Data management plans and planning\". In Pryor, G.. <i>Managing Research Data<\/i>. Facet. pp. 83\u2013104. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.29085%2F9781856048910.006\" target=\"_blank\">10.29085\/9781856048910.006<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781856048910.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+5%3A+Data+management+plans+and+planning&rft.atitle=Managing+Research+Data&rft.aulast=Donnelly%2C+M.&rft.au=Donnelly%2C+M.&rft.date=2012&rft.pages=pp.%26nbsp%3B83%E2%80%93104&rft.pub=Facet&rft_id=info:doi\/10.29085%2F9781856048910.006&rft.isbn=9781856048910&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JonesHowTo11-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JonesHowTo11_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Jones, S. (2011). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.dcc.ac.uk\/guidance\/how-guides\/develop-data-plan\" target=\"_blank\">\"How to Develop a Data Management and Sharing Plan\"<\/a>. Digital Curation Centre<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.dcc.ac.uk\/guidance\/how-guides\/develop-data-plan\" target=\"_blank\">https:\/\/www.dcc.ac.uk\/guidance\/how-guides\/develop-data-plan<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+to+Develop+a+Data+Management+and+Sharing+Plan&rft.atitle=&rft.aulast=Jones%2C+S.&rft.au=Jones%2C+S.&rft.date=2011&rft.pub=Digital+Curation+Centre&rft_id=https%3A%2F%2Fwww.dcc.ac.uk%2Fguidance%2Fhow-guides%2Fdevelop-data-plan&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO31000-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISO31000_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISO31000_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-ISO31000_4-2\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/65694.html\" target=\"_blank\">\"ISO 31000:2018 Risk management \u2014 Guidelines\"<\/a>. International Organization for Standardization. February 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/65694.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/65694.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO+31000%3A2018+Risk+management+%E2%80%94+Guidelines&rft.atitle=&rft.date=February+2018&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F65694.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISACA_DMM-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISACA_DMM_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/cmmiinstitute.com\/data-management-maturity\" target=\"_blank\">\"Data Management Maturity (DMM)\"<\/a>. Information System Audit and Control Association, Inc. 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/cmmiinstitute.com\/data-management-maturity\" target=\"_blank\">https:\/\/cmmiinstitute.com\/data-management-maturity<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 22 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+Management+Maturity+%28DMM%29&rft.atitle=&rft.date=2019&rft.pub=Information+System+Audit+and+Control+Association%2C+Inc&rft_id=https%3A%2F%2Fcmmiinstitute.com%2Fdata-management-maturity&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NewmanOver08-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NewmanOver08_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Newman, D.; Logan, D. (23 December 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.gartner.com\/en\/documents\/846312\/overview-gartner-introduces-the-eim-maturity-model\" target=\"_blank\">\"Overview: Gartner Introduces the EIM Maturity Model\"<\/a>. Gartner<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.gartner.com\/en\/documents\/846312\/overview-gartner-introduces-the-eim-maturity-model\" target=\"_blank\">https:\/\/www.gartner.com\/en\/documents\/846312\/overview-gartner-introduces-the-eim-maturity-model<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Overview%3A+Gartner+Introduces+the+EIM+Maturity+Model&rft.atitle=&rft.aulast=Newman%2C+D.%3B+Logan%2C+D.&rft.au=Newman%2C+D.%3B+Logan%2C+D.&rft.date=23+December+2008&rft.pub=Gartner&rft_id=https%3A%2F%2Fwww.gartner.com%2Fen%2Fdocuments%2F846312%2Foverview-gartner-introduces-the-eim-maturity-model&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JonesHowTo13-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JonesHowTo13_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-JonesHowTo13_7-1\">7.1<\/a><\/sup> <sup><a href=\"#cite_ref-JonesHowTo13_7-2\">7.2<\/a><\/sup> <sup><a href=\"#cite_ref-JonesHowTo13_7-3\">7.3<\/a><\/sup> <sup><a href=\"#cite_ref-JonesHowTo13_7-4\">7.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Jones, S.; Pryor, G.; Whyte, A. (25 March 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.dcc.ac.uk\/guidance\/how-guides\/how-develop-rdm-services\" target=\"_blank\">\"How to Develop RDM Services - A guide for HEIs\"<\/a>. Digital Curation Centre<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.dcc.ac.uk\/guidance\/how-guides\/how-develop-rdm-services\" target=\"_blank\">https:\/\/www.dcc.ac.uk\/guidance\/how-guides\/how-develop-rdm-services<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+to+Develop+RDM+Services+-+A+guide+for+HEIs&rft.atitle=&rft.aulast=Jones%2C+S.%3B+Pryor%2C+G.%3B+Whyte%2C+A.&rft.au=Jones%2C+S.%3B+Pryor%2C+G.%3B+Whyte%2C+A.&rft.date=25+March+2013&rft.pub=Digital+Curation+Centre&rft_id=https%3A%2F%2Fwww.dcc.ac.uk%2Fguidance%2Fhow-guides%2Fhow-develop-rdm-services&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UNESCORecords17-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UNESCORecords17_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">UNESCO (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/unesdoc.unesco.org\/ark:\/48223\/pf0000260889.page=116\" target=\"_blank\">\"Records of the General Conference, 39th session, Paris, 30 October-14 November 2017, v. 1: Resolutions\"<\/a>. p. 116<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/unesdoc.unesco.org\/ark:\/48223\/pf0000260889.page=116\" target=\"_blank\">https:\/\/unesdoc.unesco.org\/ark:\/48223\/pf0000260889.page=116<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Records+of+the+General+Conference%2C+39th+session%2C+Paris%2C+30+October-14+November+2017%2C+v.+1%3A+Resolutions&rft.atitle=&rft.aulast=UNESCO&rft.au=UNESCO&rft.date=2017&rft.pages=p.+116&rft_id=https%3A%2F%2Funesdoc.unesco.org%2Fark%3A%2F48223%2Fpf0000260889.page%3D116&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CanhosData17-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CanhosData17_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-CanhosData17_9-1\">9.1<\/a><\/sup> <sup><a href=\"#cite_ref-CanhosData17_9-2\">9.2<\/a><\/sup> <sup><a href=\"#cite_ref-CanhosData17_9-3\">9.3<\/a><\/sup> <sup><a href=\"#cite_ref-CanhosData17_9-4\">9.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Canhos, D.A.L. (2017). \"Data Management Plan: Brazil's Virtual Herbarium\". <i>RIO<\/i> <b>3<\/b>: e14675. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.3.e14675\" target=\"_blank\">10.3897\/rio.3.e14675<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+Plan%3A+Brazil%27s+Virtual+Herbarium&rft.jtitle=RIO&rft.aulast=Canhos%2C+D.A.L.&rft.au=Canhos%2C+D.A.L.&rft.date=2017&rft.volume=3&rft.pages=e14675&rft_id=info:doi\/10.3897%2Frio.3.e14675&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FeyBoulder16-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FeyBoulder16_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-FeyBoulder16_10-1\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-FeyBoulder16_10-2\">10.2<\/a><\/sup> <sup><a href=\"#cite_ref-FeyBoulder16_10-3\">10.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fey, J.; Anderson, S. (2016). \"Boulder Creek Critical Zone Observatory Data Management Plan\". <i>RIO<\/i> <b>2<\/b>: e9419. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/https%3A%2F%2Fdoi.org%2F10.3897%2Frio.2.e9419\" target=\"_blank\">https:\/\/doi.org\/10.3897\/rio.2.e9419<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Boulder+Creek+Critical+Zone+Observatory+Data+Management+Plan&rft.jtitle=RIO&rft.aulast=Fey%2C+J.%3B+Anderson%2C+S.&rft.au=Fey%2C+J.%3B+Anderson%2C+S.&rft.date=2016&rft.volume=2&rft.pages=e9419&rft_id=info:doi\/https%3A%2F%2Fdoi.org%2F10.3897%2Frio.2.e9419&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FisherAPolit16-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FisherAPolit16_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-FisherAPolit16_11-1\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-FisherAPolit16_11-2\">11.2<\/a><\/sup> <sup><a href=\"#cite_ref-FisherAPolit16_11-3\">11.3<\/a><\/sup> <sup><a href=\"#cite_ref-FisherAPolit16_11-4\">11.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fisher, J.; Nading, A.M. (2016). \"A Political Ecology of Value: A Cohort-Based Ethnography of the Environmental Turn in Nicaraguan Urban Social Policy\". <i>RIO<\/i> <b>2<\/b>: e8720. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.2.e8720\" target=\"_blank\">10.3897\/rio.2.e8720<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Political+Ecology+of+Value%3A+A+Cohort-Based+Ethnography+of+the+Environmental+Turn+in+Nicaraguan+Urban+Social+Policy&rft.jtitle=RIO&rft.aulast=Fisher%2C+J.%3B+Nading%2C+A.M.&rft.au=Fisher%2C+J.%3B+Nading%2C+A.M.&rft.date=2016&rft.volume=2&rft.pages=e8720&rft_id=info:doi\/10.3897%2Frio.2.e8720&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GattoData17-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GattoData17_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-GattoData17_12-1\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-GattoData17_12-2\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-GattoData17_12-3\">12.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gatto, L. (2017). \"Data Management Plan for a Biotechnology and Biological Sciences Research Council (BBSRC) Tools and Resources Development Fund (TRDF) Grant\". <i>RIO<\/i> <b>3<\/b>: e11624. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.3.e11624\" target=\"_blank\">10.3897\/rio.3.e11624<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+Plan+for+a+Biotechnology+and+Biological+Sciences+Research+Council+%28BBSRC%29+Tools+and+Resources+Development+Fund+%28TRDF%29+Grant&rft.jtitle=RIO&rft.aulast=Gatto%2C+L.&rft.au=Gatto%2C+L.&rft.date=2017&rft.volume=3&rft.pages=e11624&rft_id=info:doi\/10.3897%2Frio.3.e11624&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McWhorterCoast16-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-McWhorterCoast16_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-McWhorterCoast16_13-1\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-McWhorterCoast16_13-2\">13.2<\/a><\/sup> <sup><a href=\"#cite_ref-McWhorterCoast16_13-3\">13.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McWhorter, J.; Wright, D.; Thomas, J. (2016). \"Coastal Data Information Program (CDIP)\". <i>RIO<\/i> <b>2<\/b>: e8827. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.2.e8827\" target=\"_blank\">10.3897\/rio.2.e8827<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Coastal+Data+Information+Program+%28CDIP%29&rft.jtitle=RIO&rft.aulast=McWhorter%2C+J.%3B+Wright%2C+D.%3B+Thomas%2C+J.&rft.au=McWhorter%2C+J.%3B+Wright%2C+D.%3B+Thomas%2C+J.&rft.date=2016&rft.volume=2&rft.pages=e8827&rft_id=info:doi\/10.3897%2Frio.2.e8827&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NeylonData17-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NeylonData17_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-NeylonData17_14-1\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-NeylonData17_14-2\">14.2<\/a><\/sup> <sup><a href=\"#cite_ref-NeylonData17_14-3\">14.3<\/a><\/sup> <sup><a href=\"#cite_ref-NeylonData17_14-4\">14.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Neylon, C. (2017). \"Data Management Plan: IDRC Data Sharing Pilot Project\". <i>RIO<\/i> <b>3<\/b>: e14672. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.3.e14672\" target=\"_blank\">10.3897\/rio.3.e14672<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+Plan%3A+IDRC+Data+Sharing+Pilot+Project&rft.jtitle=RIO&rft.aulast=Neylon%2C+C.&rft.au=Neylon%2C+C.&rft.date=2017&rft.volume=3&rft.pages=e14672&rft_id=info:doi\/10.3897%2Frio.3.e14672&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NicholsMigrat16-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NicholsMigrat16_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-NicholsMigrat16_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-NicholsMigrat16_15-2\">15.2<\/a><\/sup> <sup><a href=\"#cite_ref-NicholsMigrat16_15-3\">15.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nichols, H.; Stolze, S. (2016). \"Migration of legacy data to new media formats for long-time storage and maximum visibility: Modern pollen data from the Canadian Arctic (1972\/1973)\". <i>RIO<\/i> <b>2<\/b>: e10269. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.2.e10269\" target=\"_blank\">10.3897\/rio.2.e10269<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Migration+of+legacy+data+to+new+media+formats+for+long-time+storage+and+maximum+visibility%3A+Modern+pollen+data+from+the+Canadian+Arctic+%281972%2F1973%29&rft.jtitle=RIO&rft.aulast=Nichols%2C+H.%3B+Stolze%2C+S.&rft.au=Nichols%2C+H.%3B+Stolze%2C+S.&rft.date=2016&rft.volume=2&rft.pages=e10269&rft_id=info:doi\/10.3897%2Frio.2.e10269&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PannellData16-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PannellData16_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-PannellData16_16-1\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-PannellData16_16-2\">16.2<\/a><\/sup> <sup><a href=\"#cite_ref-PannellData16_16-3\">16.3<\/a><\/sup> <sup><a href=\"#cite_ref-PannellData16_16-4\">16.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pannell, J.L. (2016). \"Data Management Plan for PhD Thesis \"Climatic Limitation of Alien Weeds in New Zealand: Enhancing Species Distribution Models with Field Data\"\". <i>RIO<\/i> <b>2<\/b>: e10600. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.2.e10600\" target=\"_blank\">10.3897\/rio.2.e10600<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+Plan+for+PhD+Thesis+%22Climatic+Limitation+of+Alien+Weeds+in+New+Zealand%3A+Enhancing+Species+Distribution+Models+with+Field+Data%22&rft.jtitle=RIO&rft.aulast=Pannell%2C+J.L.&rft.au=Pannell%2C+J.L.&rft.date=2016&rft.volume=2&rft.pages=e10600&rft_id=info:doi\/10.3897%2Frio.2.e10600&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TraynorData17-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TraynorData17_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-TraynorData17_17-1\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-TraynorData17_17-2\">17.2<\/a><\/sup> <sup><a href=\"#cite_ref-TraynorData17_17-3\">17.3<\/a><\/sup> <sup><a href=\"#cite_ref-TraynorData17_17-4\">17.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Traynor, C. (2017). \"Data Management Plan: Empowering Indigenous Peoples and Knowledge Systems Related to Climate Change and Intellectual Property Rights\". <i>RIO<\/i> <b>3<\/b>: e15111. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.3.e15111\" target=\"_blank\">10.3897\/rio.3.e15111<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+Plan%3A+Empowering+Indigenous+Peoples+and+Knowledge+Systems+Related+to+Climate+Change+and+Intellectual+Property+Rights&rft.jtitle=RIO&rft.aulast=Traynor%2C+C.&rft.au=Traynor%2C+C.&rft.date=2017&rft.volume=3&rft.pages=e15111&rft_id=info:doi\/10.3897%2Frio.3.e15111&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WaelData17-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WaelData17_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-WaelData17_18-1\">18.1<\/a><\/sup> <sup><a href=\"#cite_ref-WaelData17_18-2\">18.2<\/a><\/sup> <sup><a href=\"#cite_ref-WaelData17_18-3\">18.3<\/a><\/sup> <sup><a href=\"#cite_ref-WaelData17_18-4\">18.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wael, R. (2017). \"Data Management Plan: HarassMap\". <i>RIO<\/i> <b>3<\/b>: e15133. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.3.e15133\" target=\"_blank\">10.3897\/rio.3.e15133<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+Plan%3A+HarassMap&rft.jtitle=RIO&rft.aulast=Wael%2C+R.&rft.au=Wael%2C+R.&rft.date=2017&rft.volume=3&rft.pages=e15133&rft_id=info:doi\/10.3897%2Frio.3.e15133&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WhiteData16-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WhiteData16_19-0\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-WhiteData16_19-1\">19.1<\/a><\/sup> <sup><a href=\"#cite_ref-WhiteData16_19-2\">19.2<\/a><\/sup> <sup><a href=\"#cite_ref-WhiteData16_19-3\">19.3<\/a><\/sup> <sup><a href=\"#cite_ref-WhiteData16_19-4\">19.4<\/a><\/sup> <sup><a href=\"#cite_ref-WhiteData16_19-5\">19.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">White, E.P. (2016). \"Data Management Plan for Moore Investigator in Data Driven Discovery Grant\". <i>RIO<\/i> <b>2<\/b>: e10708. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.2.e10708\" target=\"_blank\">10.3897\/rio.2.e10708<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+Plan+for+Moore+Investigator+in+Data+Driven+Discovery+Grant&rft.jtitle=RIO&rft.aulast=White%2C+E.P.&rft.au=White%2C+E.P.&rft.date=2016&rft.volume=2&rft.pages=e10708&rft_id=info:doi\/10.3897%2Frio.2.e10708&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WoolfreyData17-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WoolfreyData17_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-WoolfreyData17_20-1\">20.1<\/a><\/sup> <sup><a href=\"#cite_ref-WoolfreyData17_20-2\">20.2<\/a><\/sup> <sup><a href=\"#cite_ref-WoolfreyData17_20-3\">20.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Woolfrey, L. (2017). \"Data Management Plan: Opening access to economic data to prevent tobacco related diseases in Africa\". <i>RIO<\/i> <b>3<\/b>: e14837. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.3.e14837\" target=\"_blank\">10.3897\/rio.3.e14837<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+Plan%3A+Opening+access+to+economic+data+to+prevent+tobacco+related+diseases+in+Africa&rft.jtitle=RIO&rft.aulast=Woolfrey%2C+L.&rft.au=Woolfrey%2C+L.&rft.date=2017&rft.volume=3&rft.pages=e14837&rft_id=info:doi\/10.3897%2Frio.3.e14837&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XuAData16-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-XuAData16_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-XuAData16_21-1\">21.1<\/a><\/sup> <sup><a href=\"#cite_ref-XuAData16_21-2\">21.2<\/a><\/sup> <sup><a href=\"#cite_ref-XuAData16_21-3\">21.3<\/a><\/sup> <sup><a href=\"#cite_ref-XuAData16_21-4\">21.4<\/a><\/sup> <sup><a href=\"#cite_ref-XuAData16_21-5\">21.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xu, H.; Ishida, M.; Wang, M. (2016). \"A Data Management Plan for Effects of particle size on physical and chemical properties of mine wastes\". <i>RIO<\/i> <b>2<\/b>: e11065. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3897%2Frio.2.e11065\" target=\"_blank\">10.3897\/rio.2.e11065<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Data+Management+Plan+for+Effects+of+particle+size+on+physical+and+chemical+properties+of+mine+wastes&rft.jtitle=RIO&rft.aulast=Xu%2C+H.%3B+Ishida%2C+M.%3B+Wang%2C+M.&rft.au=Xu%2C+H.%3B+Ishida%2C+M.%3B+Wang%2C+M.&rft.date=2016&rft.volume=2&rft.pages=e11065&rft_id=info:doi\/10.3897%2Frio.2.e11065&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PeisertOpen17-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PeisertOpen17_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-PeisertOpen17_22-1\">22.1<\/a><\/sup> <sup><a href=\"#cite_ref-PeisertOpen17_22-2\">22.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Peisert, S.; Welch, V.; Adams, A. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/hdl.handle.net\/2022\/21259\" target=\"_blank\">\"Open Science Cyber Risk Profile (OSCRP)\"<\/a>. <i>IUScholar Works<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/hdl.handle.net\/2022\/21259\" target=\"_blank\">http:\/\/hdl.handle.net\/2022\/21259<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Open+Science+Cyber+Risk+Profile+%28OSCRP%29&rft.atitle=IUScholar+Works&rft.aulast=Peisert%2C+S.%3B+Welch%2C+V.%3B+Adams%2C+A.+et+al.&rft.au=Peisert%2C+S.%3B+Welch%2C+V.%3B+Adams%2C+A.+et+al.&rft.date=2017&rft_id=http%3A%2F%2Fhdl.handle.net%2F2022%2F21259&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilkinsonTheFAIR16-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WilkinsonTheFAIR16_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-WilkinsonTheFAIR16_23-1\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-WilkinsonTheFAIR16_23-2\">23.2<\/a><\/sup> <sup><a href=\"#cite_ref-WilkinsonTheFAIR16_23-3\">23.3<\/a><\/sup> <sup><a href=\"#cite_ref-WilkinsonTheFAIR16_23-4\">23.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175\" target=\"_blank\">\"The FAIR Guiding Principles for scientific data management and stewardship\"<\/a>. <i>Scientific Data<\/i> <b>3<\/b>: 160018. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fsdata.2016.18\" target=\"_blank\">10.1038\/sdata.2016.18<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4792175\/\" target=\"_blank\">PMC4792175<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26978244\" target=\"_blank\">26978244<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+FAIR+Guiding+Principles+for+scientific+data+management+and+stewardship&rft.jtitle=Scientific+Data&rft.aulast=Wilkinson%2C+M.D.%3B+Dumontier%2C+M.%3B+Aalbersberg%2C+I.J.+et+al.&rft.au=Wilkinson%2C+M.D.%3B+Dumontier%2C+M.%3B+Aalbersberg%2C+I.J.+et+al.&rft.date=2016&rft.volume=3&rft.pages=160018&rft_id=info:doi\/10.1038%2Fsdata.2016.18&rft_id=info:pmc\/PMC4792175&rft_id=info:pmid\/26978244&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4792175&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FerreiraData14-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FerreiraData14_24-0\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-FerreiraData14_24-1\">24.1<\/a><\/sup> <sup><a href=\"#cite_ref-FerreiraData14_24-2\">24.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ferreira, F.; Coimbra, M.E.; Bairr\u00e3o, R. et al. (2014). \"Data Management in Metagenomics: A Risk Management Approach\". <i>International Journal of Digital Curation<\/i> <b>9<\/b> (1): 41\u201356. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2218%2Fijdc.v9i1.299\" target=\"_blank\">10.2218\/ijdc.v9i1.299<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Management+in+Metagenomics%3A+A+Risk+Management+Approach&rft.jtitle=International+Journal+of+Digital+Curation&rft.aulast=Ferreira%2C+F.%3B+Coimbra%2C+M.E.%3B+Bairr%C3%A3o%2C+R.+et+al.&rft.au=Ferreira%2C+F.%3B+Coimbra%2C+M.E.%3B+Bairr%C3%A3o%2C+R.+et+al.&rft.date=2014&rft.volume=9&rft.issue=1&rft.pages=41%E2%80%9356&rft_id=info:doi\/10.2218%2Fijdc.v9i1.299&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SynFo-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SynFo_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">University Computing Centre (Rechenzentrum) (2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.rz.uni-kiel.de\/en\/projects\/synfo-creating-synergies-on-the-operational-level-of-research-data-management\" target=\"_blank\">\"SynFo - Creating synergies on the operational level of research data management\"<\/a>. Kiel University<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.rz.uni-kiel.de\/en\/projects\/synfo-creating-synergies-on-the-operational-level-of-research-data-management\" target=\"_blank\">https:\/\/www.rz.uni-kiel.de\/en\/projects\/synfo-creating-synergies-on-the-operational-level-of-research-data-management<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SynFo+-+Creating+synergies+on+the+operational+level+of+research+data+management&rft.atitle=&rft.aulast=University+Computing+Centre+%28Rechenzentrum%29&rft.au=University+Computing+Centre+%28Rechenzentrum%29&rft.date=2019&rft.pub=Kiel+University&rft_id=https%3A%2F%2Fwww.rz.uni-kiel.de%2Fen%2Fprojects%2Fsynfo-creating-synergies-on-the-operational-level-of-research-data-management&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSIITGrund15-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BSIITGrund15_27-0\">26.00<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-1\">26.01<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-2\">26.02<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-3\">26.03<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-4\">26.04<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-5\">26.05<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-6\">26.06<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-7\">26.07<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-8\">26.08<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-9\">26.09<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-10\">26.10<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-11\">26.11<\/a><\/sup> <sup><a href=\"#cite_ref-BSIITGrund15_27-12\">26.12<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">German Federal Office for Information Security (22 December 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20200128211607\/https:\/\/www.bsi.bund.de\/SharedDocs\/Downloads\/DE\/BSI\/Grundschutz\/International\/GSK_15_EL_EN_Draft.html\" target=\"_blank\">\"IT-Grundschutz-catalogues 15th version - 2015 (Draft)\"<\/a>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bsi.bund.de\/SharedDocs\/Downloads\/DE\/BSI\/Grundschutz\/International\/GSK_15_EL_EN_Draft.html\" target=\"_blank\">the original<\/a> on 28 January 2020<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20200128211607\/https:\/\/www.bsi.bund.de\/SharedDocs\/Downloads\/DE\/BSI\/Grundschutz\/International\/GSK_15_EL_EN_Draft.html\" target=\"_blank\">https:\/\/web.archive.org\/web\/20200128211607\/https:\/\/www.bsi.bund.de\/SharedDocs\/Downloads\/DE\/BSI\/Grundschutz\/International\/GSK_15_EL_EN_Draft.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=IT-Grundschutz-catalogues+15th+version+-+2015+%28Draft%29&rft.atitle=&rft.aulast=German+Federal+Office+for+Information+Security&rft.au=German+Federal+Office+for+Information+Security&rft.date=22+December+2016&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20200128211607%2Fhttps%3A%2F%2Fwww.bsi.bund.de%2FSharedDocs%2FDownloads%2FDE%2FBSI%2FGrundschutz%2FInternational%2FGSK_15_EL_EN_Draft.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CollinsTurning18-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CollinsTurning18_28-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-CollinsTurning18_28-1\">27.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Collins, S.; Genova, F.; Harrower, N. (26 November 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/op.europa.eu\/en\/publication-detail\/-\/publication\/7769a148-f1f6-11e8-9982-01aa75ed71a1\/language-en\" target=\"_blank\">\"Turning FAIR into reality\"<\/a>. European Commission. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2777%2F1524\" target=\"_blank\">10.2777\/1524<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/op.europa.eu\/en\/publication-detail\/-\/publication\/7769a148-f1f6-11e8-9982-01aa75ed71a1\/language-en\" target=\"_blank\">https:\/\/op.europa.eu\/en\/publication-detail\/-\/publication\/7769a148-f1f6-11e8-9982-01aa75ed71a1\/language-en<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Turning+FAIR+into+reality&rft.atitle=&rft.aulast=Collins%2C+S.%3B+Genova%2C+F.%3B+Harrower%2C+N.&rft.au=Collins%2C+S.%3B+Genova%2C+F.%3B+Harrower%2C+N.&rft.date=26+November+2018&rft.pub=European+Commission&rft_id=info:doi\/10.2777%2F1524&rft_id=https%3A%2F%2Fop.europa.eu%2Fen%2Fpublication-detail%2F-%2Fpublication%2F7769a148-f1f6-11e8-9982-01aa75ed71a1%2Flanguage-en&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ELEXReg511-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ELEXReg511_30-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-ELEXReg511_30-1\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/data.europa.eu\/eli\/reg\/2014\/511\/oj\" target=\"_blank\">\"Regulation (EU) No 511\/2014 of the European Parliament and of the Council of 16 April 2014 on compliance measures for users from the Nagoya Protocol on Access to Genetic Resources and the Fair and Equitable Sharing of Benefits Arising from their Utilization in the Union Text with EEA relevance\"<\/a>. <i>EUR-Lex<\/i>. European Union. 16 April 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/data.europa.eu\/eli\/reg\/2014\/511\/oj\" target=\"_blank\">http:\/\/data.europa.eu\/eli\/reg\/2014\/511\/oj<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 December 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Regulation+%28EU%29+No+511%2F2014+of+the+European+Parliament+and+of+the+Council+of+16+April+2014+on+compliance+measures+for+users+from+the+Nagoya+Protocol+on+Access+to+Genetic+Resources+and+the+Fair+and+Equitable+Sharing+of+Benefits+Arising+from+their+Utilization+in+the+Union+Text+with+EEA+relevance&rft.atitle=EUR-Lex&rft.date=16+April+2014&rft.pub=European+Union&rft_id=http%3A%2F%2Fdata.europa.eu%2Feli%2Freg%2F2014%2F511%2Foj&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-vanVegchelImplem18-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-vanVegchelImplem18_31-0\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-vanVegchelImplem18_31-1\">29.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/publications.vo.eu\/implementation-of-nagoya-protocol\/\" target=\"_blank\">\"Implementation of Nagoya Protocol: A comparison between The Netherlands, Belgium and Germany\"<\/a>. <i>vo.eu<\/i>. 18 June 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/publications.vo.eu\/implementation-of-nagoya-protocol\/\" target=\"_blank\">https:\/\/publications.vo.eu\/implementation-of-nagoya-protocol\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Implementation+of+Nagoya+Protocol%3A+A+comparison+between+The+Netherlands%2C+Belgium+and+Germany&rft.atitle=vo.eu&rft.date=18+June+2018&rft_id=https%3A%2F%2Fpublications.vo.eu%2Fimplementation-of-nagoya-protocol%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CCFaq-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CCFaq_33-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-CCFaq_33-1\">30.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/creativecommons.org\/faq\/\" target=\"_blank\">\"Frequently Asked Questions\"<\/a>. <i>Creative Commons<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/creativecommons.org\/faq\/\" target=\"_blank\">https:\/\/creativecommons.org\/faq\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 December 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Frequently+Asked+Questions&rft.atitle=Creative+Commons&rft_id=https%3A%2F%2Fcreativecommons.org%2Ffaq%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LipinskiLib12-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LipinskiLib12_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Lipinski, T.A. (2012). <i>Librarian's Legal Companion for Licensing Information Resources and Legal Services<\/i>. Neal-Schuman Publishers. p. 312. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781555706104.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Librarian%27s+Legal+Companion+for+Licensing+Information+Resources+and+Legal+Services&rft.aulast=Lipinski%2C+T.A.&rft.au=Lipinski%2C+T.A.&rft.date=2012&rft.pages=p.%26nbsp%3B312&rft.pub=Neal-Schuman+Publishers&rft.isbn=9781555706104&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AP29Adequ18-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AP29Adequ18_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Article 29 Data Protection Working Party (06 February 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ec.europa.eu\/newsroom\/article29\/item-detail.cfm?item_id=614108\" target=\"_blank\">\"Working document on Adequacy Referential (wp254rev.01)\"<\/a>. European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ec.europa.eu\/newsroom\/article29\/item-detail.cfm?item_id=614108\" target=\"_blank\">https:\/\/ec.europa.eu\/newsroom\/article29\/item-detail.cfm?item_id=614108<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 December 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Working+document+on+Adequacy+Referential+%28wp254rev.01%29&rft.atitle=&rft.aulast=Article+29+Data+Protection+Working+Party&rft.au=Article+29+Data+Protection+Working+Party&rft.date=06+February+2018&rft.pub=European+Commission&rft_id=https%3A%2F%2Fec.europa.eu%2Fnewsroom%2Farticle29%2Fitem-detail.cfm%3Fitem_id%3D614108&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ECAdeq19-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ECAdeq19_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">European Commission (14 January 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ec.europa.eu\/info\/law\/law-topic\/data-protection\/international-dimension-data-protection\/adequacy-decisions_en\" target=\"_blank\">\"Adequacy decisions\"<\/a>. European Commission<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ec.europa.eu\/info\/law\/law-topic\/data-protection\/international-dimension-data-protection\/adequacy-decisions_en\" target=\"_blank\">https:\/\/ec.europa.eu\/info\/law\/law-topic\/data-protection\/international-dimension-data-protection\/adequacy-decisions_en<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 December 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Adequacy+decisions&rft.atitle=&rft.aulast=European+Commission&rft.au=European+Commission&rft.date=14+January+2019&rft.pub=European+Commission&rft_id=https%3A%2F%2Fec.europa.eu%2Finfo%2Flaw%2Flaw-topic%2Fdata-protection%2Finternational-dimension-data-protection%2Fadequacy-decisions_en&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ELEXReg679-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ELEXReg679_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/data.europa.eu\/eli\/reg\/2016\/679\/2016-05-04\" target=\"_blank\">\"Consolidated text: Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation) (Text with EEA relevance)\"<\/a>. <i>EUR-Lex<\/i>. European Union. 27 April 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/data.europa.eu\/eli\/reg\/2016\/679\/2016-05-04\" target=\"_blank\">http:\/\/data.europa.eu\/eli\/reg\/2016\/679\/2016-05-04<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Consolidated+text%3A+Regulation+%28EU%29+2016%2F679+of+the+European+Parliament+and+of+the+Council+of+27+April+2016+on+the+protection+of+natural+persons+with+regard+to+the+processing+of+personal+data+and+on+the+free+movement+of+such+data%2C+and+repealing+Directive+95%2F46%2FEC+%28General+Data+Protection+Regulation%29+%28Text+with+EEA+relevance%29&rft.atitle=EUR-Lex&rft.date=27+April+2016&rft.pub=European+Union&rft_id=http%3A%2F%2Fdata.europa.eu%2Feli%2Freg%2F2016%2F679%2F2016-05-04&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ReichLOCKSS09-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ReichLOCKSS09_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Reich, V.; Rosenthal, D.S.H. (2009). \"LOCKSS (Lots of Copies Keep Stuff Safe)\". <i>New Review of Academic Librarianship<\/i> <b>6<\/b> (1): 155\u201361. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F13614530009516806\" target=\"_blank\">10.1080\/13614530009516806<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LOCKSS+%28Lots+of+Copies+Keep+Stuff+Safe%29&rft.jtitle=New+Review+of+Academic+Librarianship&rft.aulast=Reich%2C+V.%3B+Rosenthal%2C+D.S.H.&rft.au=Reich%2C+V.%3B+Rosenthal%2C+D.S.H.&rft.date=2009&rft.volume=6&rft.issue=1&rft.pages=155%E2%80%9361&rft_id=info:doi\/10.1080%2F13614530009516806&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PengTheRepro15-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PengTheRepro15_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Peng, R. (2015). \"The reproducibility crisis in science: A statistical counterattack\". <i>Sginificance<\/i> <b>12<\/b> (3): 30\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1740-9713.2015.00827.x\" target=\"_blank\">10.1111\/j.1740-9713.2015.00827.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+reproducibility+crisis+in+science%3A+A+statistical+counterattack&rft.jtitle=Sginificance&rft.aulast=Peng%2C+R.&rft.au=Peng%2C+R.&rft.date=2015&rft.volume=12&rft.issue=3&rft.pages=30%E2%80%9332&rft_id=info:doi\/10.1111%2Fj.1740-9713.2015.00827.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Jim.C3.A9nezFour17-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Jim.C3.A9nezFour17_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jim\u00e9nez, R.C.; Kuzak, M; Alhamdoosh, M. et al. (2017). \"Four simple recommendations to encourage best practices in research software [version 1; peer review: 3 approved]\". <i>F1000Research<\/i> <b>6<\/b>: 876. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.12688%2Ff1000research.11407.1\" target=\"_blank\">10.12688\/f1000research.11407.1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Four+simple+recommendations+to+encourage+best+practices+in+research+software+%5Bversion+1%3B+peer+review%3A+3+approved%5D&rft.jtitle=F1000Research&rft.aulast=Jim%C3%A9nez%2C+R.C.%3B+Kuzak%2C+M%3B+Alhamdoosh%2C+M.+et+al.&rft.au=Jim%C3%A9nez%2C+R.C.%3B+Kuzak%2C+M%3B+Alhamdoosh%2C+M.+et+al.&rft.date=2017&rft.volume=6&rft.pages=876&rft_id=info:doi\/10.12688%2Ff1000research.11407.1&rfr_id=info:sid\/en.wikipedia.org:Journal:Towards_a_risk_catalog_for_data_management_plans\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; this version lists them in order of appearance, by design.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194109\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.633 seconds\nReal time usage: 0.652 seconds\nPreprocessor visited node count: 27709\/1000000\nPreprocessor generated node count: 42375\/1000000\nPost\u2010expand include size: 175584\/2097152 bytes\nTemplate argument size: 61696\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 584.673 1 - -total\n 80.60% 471.248 2 - Template:Reflist\n 67.43% 394.243 37 - Template:Citation\/core\n 39.49% 230.877 19 - Template:Cite_journal\n 27.00% 157.833 16 - Template:Cite_web\n 6.37% 37.237 1 - Template:Infobox_journal_article\n 6.08% 35.566 1 - Template:Infobox\n 4.47% 26.127 2 - Template:Cite_book\n 4.20% 24.557 80 - Template:Infobox\/row\n 4.14% 24.192 25 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12407-0!*!0!!en!*!* and timestamp 20210429194108 and revision id 41868\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Towards_a_risk_catalog_for_data_management_plans\">https:\/\/www.limswiki.org\/index.php\/Journal:Towards_a_risk_catalog_for_data_management_plans<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","00bcc6c235846f18221f6a618b37957c_images":[],"00bcc6c235846f18221f6a618b37957c_timestamp":1619725268,"79a0829b6677ab7c03477c2b7c49428a_type":"article","79a0829b6677ab7c03477c2b7c49428a_title":"Making data and workflows findable for machines (Weigel et al. 2020)","79a0829b6677ab7c03477c2b7c49428a_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Making_data_and_workflows_findable_for_machines","79a0829b6677ab7c03477c2b7c49428a_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Making data and workflows findable for machines\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nMaking data and workflows findable for machinesJournal\n \nData IntelligenceAuthor(s)\n \nWeigel, Tobias; Schwardmann, Ulrich; Klump, Jens; Bendoukha, Sofiane; Quick, RobertAuthor affiliation(s)\n \nDeutsches Klimarechenzentrum, Gesellschaft f\u00fcr wissenschaftliche Datenverarbeitung G\u00f6ttingen,\r\nCSIRO, Indiana University BloomingtonPrimary contact\n \nEmail: weigel at dkrz dot deYear published\n \n2020Volume and issue\n \n2(1\u20132)Page(s)\n \n40-46DOI\n \n10.1162\/dint_a_00026ISSN\n \n2641-435XDistribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mitpressjournals.org\/doi\/full\/10.1162\/dint_a_00026Download\n \nhttps:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/dint_a_00026 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Essential requirements for automating data and workflow findability for machines \n4 Elements of a possible solution \n5 Extending capabilities with machine learning \n6 Outlook and conclusions \n7 Acknowledgements \n\n7.1 Author contributions \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nResearch data currently face a huge increase of data objects, with an increasing variety of types (data types, formats) and variety of workflows by which objects need to be managed across their lifecycle by data infrastructures. Researchers desire to shorten the workflows from data generation to analysis and publication, and the full workflow needs to become transparent to multiple stakeholders, including research administrators and funders. This poses challenges for research infrastructures and user-oriented data services in terms of not only making data and workflows findable, accessible, interoperable, and reusable (FAIR), but also doing so in a way that leverages machine support for better efficiency. One primary need yet to be addressed is that of findability, and achieving better findability has benefits for other aspects of data and workflow management. In this article, we describe how machine capabilities can be extended to make workflows more findable, in particular by leveraging the Digital Object Architecture, common object operations, and machine learning techniques.\nKeywords: findability, workflows, automation, FAIR data, data infrastructures, data services\n\nIntroduction \nIn several scientific disciplines, the number, size, and variety of data objects to be managed are growing. Examples of particular interest to the challenges discussed in this article include climate modeling[1], geophysics[2], and \u201comics\u201d-based scientific approaches.[3] The supporting data infrastructures and services are challenged to offer adequate solutions, and researchers are looking toward increased automation in their processes to cope with the needs. Aspects of automation are intrinsic to making data and workflows findable, accessible, interoperable, and reusable according to the FAIR guiding principles.[4] This article highlights the automation steps that are required to automatically identify data objects, associate them with metadata, and make both that data and the processes that generated them more findable. Persistent identifiers, machine processes with autonomous decision-making capability, and machine-actionable metadata are critical elements for practical solutions.\nThe motivation is given through the increased interest by researchers and funders in making not only those data available that underpin analysis in scientific publications, but also give insight into the generative history of these data while they were generated, processed, analyzed, and eventually published. Readers wish to investigate the provenance of data underlying publications, gaining access to contextual information on data in the provenance graph and on workflows or individual data processing steps. In this article, we investigate how such information can be aggregated and leveraged to improve the general findability of data and the workflows that produce them, improving the quality of information that search catalogs such as B2FIND or the CSIRO Data Access Portal can depend upon. The potential next step\u2014to enable machines to find resources automatically as part of orchestration\u2014will only be touched upon marginally. Concerning aggregation for findability, the article highlights key requirements and elements of possible solutions that can inform future work.\nResearchers who work with data are also interested in making their workflows more efficient, shortening the time from data production to analysis, but also short-cutting workflows, for example, when using in-situ visualization in a high-performance computing (HPC) workflow to detect errors already made during a computing run and restarting the process quickly with modified parameters. Another important usage trend is the motivation of users to work with data at higher levels of abstraction. Researchers are increasingly relying on tools such as Jupyter Notebook and standard software libraries to deal with issues of data access and management, giving rise to the wider adoption of virtual research environments (VREs).[5][6] It is much more efficient to let them focus on the scientific questions surrounding data analysis, and reduce the amount of resources they spend on data management and access. This is part of a larger cultural change\u2014which has wide impact on the evolution of data services\u2014and improving findability is a key concern.\nA key capability necessary to support future scenarios is, therefore, support at the data infrastructure level for better automation of the processes dealing with data and workflows. Out of the many possible facets related to this challenge that could be derived from the FAIR principles, we focus on the automation of findability (principles F1-F3), emphasizing that identifiers are a foundational element from which the other principles must follow.[7] A key question is: How can automated processes help to make more data and workflows findable, particularly from early research workflow stages? In this article, we understand an automated process as one that is capable of limited, autonomous decision-making. This is driven by rule systems specified by humans, but could also, in a later evolution, be replaced by means of machine learning.\n\nEssential requirements for automating data and workflow findability for machines \nTo support machine-actionable processes in data infrastructures and VREs, objects (including data and workflows, but possibly also other artifacts) need to be persistently identifiable, independent of location (F1).[7] This is the primary prerequisite for any other benefits to be realized. An important constraint is that the future preservation status of an object is likely unclear at early stages, but identification is nonetheless required. This shapes the choice of persistent identifier (PID) systems to employ.\nMoreover, offering elemental operations on objects, independent of their location, readily supports the needs of data management processes within data infrastructures. We can define two levels of such operations. The first level includes \u201ccreate, read, update. and delete\u201d (CRUD) operations on single objects and collections, as well as directly related support operations such as retrieving object metadata, which is critical to facilitate findability. The second level consists of more complex operations such as creating a replica (physically copy an object, then update associated metadata to describe the replica) or creating a success or new version (create a new object, the describe the relation with the predecessor in metadata).\nIn order to record a fundamental level of provenance from data processing in VREs, persistent identification must be complemented by additional metadata (F2). Following the W3C provenance (PROV) data model [8], the first action is to record the link between input and output data using wasDerivedFrom relations between entities. Extensions may then add links to the processing script or workflow (activity) and links to the executing user (agent). Finally, to enable automated processes to not only use such information (which can be ensured by relying on PROV-compatible encodings) but also discover it in the first place, the methods by which to access it must be well-defined (A1). This does not necessarily mean that the information has to be centrally stored. It could also be federated, but the mechanism has to be straightforward to use by automated processes, and the use of the identifier to access metadata according to A1 is a primary prerequisite.\nOne particular aspect for data processing is that workflows may be defined by users, but those worflows will be best used by automated processes as (web) services, i.e., they must have a machine-interpretable service description and standardized interfaces, and give back unambiguous information about execution results. A mechanism that makes such information readily available for automated agents is required.\nOne well-known established approach for addressing concerns of reproducibility, automation, and provenance, in particular, are scientific workflow systems.[9] These have seen larger adoption in the \u201comics\u201d-based research area, but they are less adopted for climate or geophysics data processing scenarios, in contrast to the adoption of interactive Python via Jupyter Notebooks. One contributing factor may be that interactive notebooks support both repetitive tasks and exploratory work modes. Exploration is, for instance, a major factor for geophysics use cases in the Scientific Software Solution Centre (SSSC)[2], where the set of possible input data is relatively small, but there is a large variation in algorithms to evaluate. Cases like this make the Jupyter Notebook particularly attractive. For HPC environments and data infrastructures, integration with fairly heavy-weight workflow systems poses a huge challenge that is often refrained from due to the large investment required. Nonetheless, the controlled environment of a workflow system has a certain appeal for automating metadata capture in the background (F2, R1), yet in view of the tremendous user adoption of interactive notebooks, a solution should potentially be suitable for both approaches.\nFinally, a few other general requirements will be critical to ensure practical success of any solution. It should be resilient against operational incidents, such as temporary server outages, and either recover from them without human intervention or fully fail over, ensuring continuous operation. While this may seem a generally good requirement for any technical system, it is even more critically so if the system is built on automated processes capable of limited autonomy.\n\nElements of a possible solution \nA consistent implementation that improves findability for machines can be broken down into several parts, following the research workflow. Ultimately, findability depends on trustworthy, gapless metadata, and improving the processes in the workflow contributes to better findability at the consumer's end. One important constraint is that even a full solution will still be semi-automatic, i.e., a human user will still need to be involved as the ultimate referee\u2014a contributor of some metadata elements that only a human can define\u2014and to ensure overall quality control.\nThe persistent identification of objects and reciprocal association (F3) with machine-interpretable metadata can be facilitated by employing the Digital Object Architecture (DOA)[10], PID Kernel Information[11], and Data Type Registries[12][13], while also following the model of FAIR Digital Objects.[14]\nIn the following, we describe along a generalized workflow how these elements can be combined to address the requirements. A key aspect is that the PID is seen as the primary anchor or \u201centry point\u201d for any agent interacting with an object, making availability and proper maintenance of PIDs a necessary pre-condition.\nAs a first step, any object should receive a PID. At the earliest workflow stages, descriptive metadata are likely not available yet, and the long-term preservation status of an object is also unclear. Therefore, a PID system that does not mandate specific metadata elements and does not enforce strong policies, such as the Handle System, is a good fit. Even more important than the choice of PID system is that, in order to support lateroperations and autonomous handling of objects, the PID should be embedded in the object. For instance, if the object is a file with a format supporting embedded metadata, the PID should be included.\nWhile descriptive metadata may not be available, support for generalized CRUD operations requires essential structural and administrative metadata to be captured, stored, and made available for requestors. Metadata capture must be highly automated and reliable, both in terms of technical reliability and ensured metadata quality. This requires an approach that may be very different from established procedures. For example, in the case of adoption by the Earth System Grid Federation (ESGF) for climate data, it became clear early on that technical solutions must be embedded in processes agreed upon by all stakeholders (users, project and data managers, infrastructure providers, and administrators), and that defining and establishing these processes is a prerequisite for subsequent technical development. This leads to a general observation that, in particular, the quality of metadata may be controlled by technical means, but high quality can only be achieved if the processes are supported by all stakeholders.\nMetadata delivery must work with low latency and in highly standardized, machine-interpretable encodings. While originally addressed toward encoding provenance, the concept of PID Kernel Information and its underlying principles can fulfill these additional requirements also to support metadata required for CRUD operations. At later workflow stages of data publication and wide sharing, PID Kernel Information is unsuitable to feed search catalogs with meaningful descriptive metadata, a process that must be complemented with other sources. Here, it may be possible to leverage existing workflows driven by the need for descriptive metadata for purposes of archival and credit-giving.\nCommon operations on objects may best be implemented according to a comprehensive specification. In the DOA framework, a Digital Object Interface Protocol (DOIP) has been defined to additionally incorporate such operation specifications. Implementing such a protocol on top of not only a single repository, but also as part of a service-oriented architecture may be more difficult, since it is likely that any single operation is offered by multiple middleware services, and that execution of an operation may have side effects or require actions to be taken by other services. A practical obstacle for implementation is the required compatibility with existing architectural components, protocols, and interfaces (e.g., REST), and fitness for use within distributed systems, where no single control point may exist to coordinate the execution of operations.\nServices and VREs for data analysis and processing such as the Scientific Software Solution Centre (SSSC)[2] or the ENES Climate Analytics Service (ECAS)[15] present a unique opportunity to implement a solution at small scale in a relatively closed environment, since their supported workflows are a smaller but representative subset of the more general research data workflow, and the central control over the VRE workflows makes implementation easier compared to distributed middleware in larger data infrastructures. Implementing automated PID assignment, metadata generation and provenance capture, and elemental object operations in a VRE may easily demonstrate improvements to downstream findability that can inform decisions on implementations in larger infrastructures.\n\nExtending capabilities with machine learning \nWe will briefly highlight two opportunities for a solution to employ machine learning techniques, concerning classification for search catalogs (F4) and building recommender systems. While the components mentioned so far can improve metadata acquisition, it is likely that gaps will remain that also cannot be covered through increased human intervention. Machine learning may help by classifying artifacts based on incomplete information, possibly also using unstructured sources such as log files from executing computing jobs or running processing tools. This may work particularly well if a VRE or scientific workflow management system is used, but may also work well in the back of common HPC jobs.\nThe most important constraint is that the result of such classification by machine learning algorithms will bear an intrinsic uncertainty. It should therefore not be a full alternative to metadata acquisition, particularly in view of the level of precision required for data preservation, but it can fuel search catalogs, as a level of uncertainty may be tolerable. Information both out of metadata and algorithmic classification may then be used to power recommender systems that enhance search catalog capabilities.[16] They may recommend, for example, input data sets or workflows for reuse to VRE users, and thus contribute to improved findability from the consumer's end.\n\nOutlook and conclusions \nWe have touched upon important requirements and key elements of a solution to improve findability of data and workflows by leveraging automation capabilities during the research workflow. Future work on the topic may derive more concrete recommendations and build demonstrators. The approach described, motivated by requirements seen in several disciplines, hints at promising solutions that could emerge out of collaborative work across disciplinary infrastructures and service providers.\nIn the end, a decisive take on automation for findability can be of benefit to multiple stakeholders. Researchers producing data can spend less time on data management and documentation, researchers reusing data and workflows will have access to metadata on a wider range of objects, and research administrators and funders may benefit from deeper insight into the impact of data-generating workflows. The wider adoption of a solution may also benefit other aspects of FAIR, e.g., interoperability and reusability.\n\nAcknowledgements \nAuthor contributions \nAll authors have made meaningful and valuable contributions in revising and proofreading the resulting manuscript. Tobias Weigel has led the editorial process.\n\nReferences \n\n\n\u2191 Balaji, V.; Taylor, K.E.; Juckes, M. et al. (2018). \"Requirements for a global data infrastructure in support of CMIP6\". Geoscientific Model Development 11 (9): 3659\u20133680. doi:10.5194\/gmd-11-3659-2018.   \n\n\u2191 2.0 2.1 2.2 Squire, G.; Wu, M.; Friedrich, C. et al. (2018). \"IN43C-0903: Scientific Software Solution Centre for Discovering, Sharing and Reusing Research Software\". Proceedings from the 2018 AGU Fall Meeting. https:\/\/agu.confex.com\/agu\/fm18\/meetingapp.cgi\/Paper\/459873 .   \n\n\u2191 Goble, C.; Cohen=Boulakia, S.; Soiland-Reyes, S. et al. (2020). \"FAIR Computational Workflows\". Data Intelligence 2 (1\u20132): 108\u201321. doi:10.1162\/dint_a_00033.   \n\n\u2191 Mons, B.; Neylon, C.; Velterop, J. et al. (2017). \"Cloudy, increasingly FAIR; revisiting the FAIR Data guiding principles for the European Open Science Cloud\". Information Services & Use 37 (1): 49\u201356. doi:10.3233\/ISU-170824.   \n\n\u2191 Wyborn, L.A.; Fraser, R.; Evans, B.J.K. et al. (2017). \"ED32B-03: Building a Generic Virtual Research Environment Framework for Multiple Earth and Space Science Domains and a Diversity of Users\". Proceedings from the 2017 AGU Fall Meeting. https:\/\/agu.confex.com\/agu\/fm17\/meetingapp.cgi\/Paper\/293857 .   \n\n\u2191 Barker, M.; Olabarriaga, S.D.; Wilkins-Diehr, N. et al. (2019). \"The global impact of science gateways, virtual research environments and virtual laboratories\". Future Generation Computer Systems 95: 240\u201348. doi:10.1016\/j.future.2018.12.026.   \n\n\u2191 7.0 7.1 Juty, N.; Wimalaratne, S.M.; Soiland-Reyes, S. et al. (2020). \"Unique, Persistent, Resolvable: Identifiers as the Foundation of FAIR\". Data Intelligence 2 (1\u20132): 30\u201339. doi:10.1162\/dint_a_00025.   \n\n\u2191 Belhajjame, K. B'Far, R.; Cheney, J. et al. (30 April 2013). \"PROV-DM: The PROV Data Model\". w3.org. https:\/\/www.w3.org\/TR\/2013\/REC-prov-dm-20130430\/ .   \n\n\u2191 Taylor, I.J.; Deelman, E.; Gannon, D.B. et al., ed. (2007). Workflows fo e-Science. Springer. ISBN 9781846287572.   \n\n\u2191 Kahn, R.; Wilensky, R. (2006). \"A framework for distributed digital object services\". International Journal on Digital Libraries 6: 115\u201323. doi:10.1007\/s00799-005-0128-x.   \n\n\u2191 Weigel, T.; Plale, B.; Parsons, M. et al. (2018). \"RDA Recommendation on PID Kernel Information\". Research Data Alliance. doi:10.15497\/rda00031. https:\/\/www.rd-alliance.org\/group\/pid-kernel-information-wg\/outcomes\/recommendation-pid-kernel-information .   \n\n\u2191 Lannom, L.; Broeder, D.; Manepalli, G. (2015). \"Data Type Registries working group output\". Research Data Alliance. doi:10.15497\/A5BCD108-ECC4-41BE-91A7-20112FF77458. https:\/\/www.rd-alliance.org\/group\/data-type-registries-wg\/outcomes\/data-type-registries .   \n\n\u2191 Schwardmann, U. (2016). \"Automated schema extraction for PID information types\". Proceedings of the 2016 IEEE International Conference on Big Data: 3036\u20133044. doi:10.1109\/BigData.2016.7840957.   \n\n\u2191 European Commission (2018). Turning FAIR Into Reality. European Commission. doi:10.2777\/1524. ISBN 9789279965463.   \n\n\u2191 Bendoukha, S.; Weigel, T.; Fiore, S. et al. (2018). \"ENES Climate Analytics Service (ECAS)\" (PDF). Geophysical Researh Abstracts 20 (EGU2018-12549). https:\/\/meetingorganizer.copernicus.org\/EGU2018\/EGU2018-12549.pdf .   \n\n\u2191 Devaraju, A.; Berkovsky, S. (2018). \"A Hybrid Recommendation Approach for Open Research Datasets\". Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization: 207\u201311. doi:10.1145\/3209219.3209250.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Making_data_and_workflows_findable_for_machines\">https:\/\/www.limswiki.org\/index.php\/Journal:Making_data_and_workflows_findable_for_machines<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on data infrastructureLIMSwiki journal articles on data management and sharingLIMSwiki journal articles on research\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 31 January 2021, at 21:12.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 170 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","79a0829b6677ab7c03477c2b7c49428a_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Making_data_and_workflows_findable_for_machines skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Making data and workflows findable for machines<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Research\" title=\"Research\" class=\"wiki-link\" data-key=\"409634fd90113f119362927fe222f549\">Research<\/a> data currently face a huge increase of data objects, with an increasing variety of types (data types, formats) and variety of <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a> by which objects need to be managed across their lifecycle by data infrastructures. Researchers desire to shorten the workflows from data generation to <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analysis<\/a> and publication, and the full workflow needs to become transparent to multiple stakeholders, including research administrators and funders. This poses challenges for research infrastructures and user-oriented data services in terms of not only making data and workflows findable, accessible, interoperable, and reusable (<a href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\" title=\"Journal:The FAIR Guiding Principles for scientific data management and stewardship\" class=\"wiki-link\" data-key=\"e5903ddcc7734415af1d91fcd258da90\">FAIR<\/a>), but also doing so in a way that leverages machine support for better efficiency. One primary need yet to be addressed is that of findability, and achieving better findability has benefits for other aspects of data and workflow management. In this article, we describe how machine capabilities can be extended to make workflows more findable, in particular by leveraging the Digital Object Architecture, common object operations, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Machine_learning\" title=\"Machine learning\" class=\"wiki-link\" data-key=\"79aab39cfa124c958cd1dbcab3dde122\">machine learning<\/a> techniques.\n<\/p><p><b>Keywords<\/b>: findability, workflows, automation, FAIR data, data infrastructures, data services\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>In several scientific disciplines, the number, size, and variety of data objects to be managed are growing. Examples of particular interest to the challenges discussed in this article include climate modeling<sup id=\"rdp-ebb-cite_ref-BalajiRequire18_1-0\" class=\"reference\"><a href=\"#cite_note-BalajiRequire18-1\">[1]<\/a><\/sup>, geophysics<sup id=\"rdp-ebb-cite_ref-SquireScient18_2-0\" class=\"reference\"><a href=\"#cite_note-SquireScient18-2\">[2]<\/a><\/sup>, and \u201comics\u201d-based scientific approaches.<sup id=\"rdp-ebb-cite_ref-GobleFAIR20_3-0\" class=\"reference\"><a href=\"#cite_note-GobleFAIR20-3\">[3]<\/a><\/sup> The supporting <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data infrastructures and services<\/a> are challenged to offer adequate solutions, and researchers are looking toward increased automation in their processes to cope with the needs. Aspects of automation are intrinsic to making data and <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a> findable, accessible, interoperable, and reusable according to the <a href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_FAIR_Guiding_Principles_for_scientific_data_management_and_stewardship\" title=\"Journal:The FAIR Guiding Principles for scientific data management and stewardship\" class=\"wiki-link\" data-key=\"e5903ddcc7734415af1d91fcd258da90\">FAIR guiding principles<\/a>.<sup id=\"rdp-ebb-cite_ref-MonsCloudy17_4-0\" class=\"reference\"><a href=\"#cite_note-MonsCloudy17-4\">[4]<\/a><\/sup> This article highlights the automation steps that are required to automatically identify data objects, associate them with <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a>, and make both that data and the processes that generated them more findable. Persistent identifiers, machine processes with autonomous decision-making capability, and machine-actionable metadata are critical elements for practical solutions.\n<\/p><p>The motivation is given through the increased interest by researchers and funders in making not only those data available that underpin <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analysis<\/a> in scientific publications, but also give insight into the generative history of these data while they were generated, processed, analyzed, and eventually published. Readers wish to investigate the provenance of data underlying publications, gaining access to contextual <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> on data in the provenance graph and on workflows or individual data processing steps. In this article, we investigate how such information can be aggregated and leveraged to improve the general findability of data and the workflows that produce them, improving the quality of information that search catalogs such as <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.eudat.eu\/services\/b2find\" target=\"_blank\">B2FIND<\/a> or the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/data.csiro.au\/dap\/home\" target=\"_blank\">CSIRO Data Access Portal<\/a> can depend upon. The potential next step\u2014to enable machines to find resources automatically as part of orchestration\u2014will only be touched upon marginally. Concerning aggregation for findability, the article highlights key requirements and elements of possible solutions that can inform future work.\n<\/p><p>Researchers who work with data are also interested in making their workflows more efficient, shortening the time from data production to analysis, but also short-cutting workflows, for example, when using <i>in-situ<\/i> visualization in a high-performance computing (HPC) workflow to detect errors already made during a computing run and restarting the process quickly with modified parameters. Another important usage trend is the motivation of users to work with data at higher levels of abstraction. Researchers are increasingly relying on tools such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Jupyter_Notebook\" title=\"Jupyter Notebook\" class=\"wiki-link\" data-key=\"26fd35430c10e009a142bbab5dbf617a\">Jupyter Notebook<\/a> and standard software libraries to deal with issues of data access and management, giving rise to the wider adoption of virtual research environments (VREs).<sup id=\"rdp-ebb-cite_ref-WybornBuild17_5-0\" class=\"reference\"><a href=\"#cite_note-WybornBuild17-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BarkerTheGlob19_6-0\" class=\"reference\"><a href=\"#cite_note-BarkerTheGlob19-6\">[6]<\/a><\/sup> It is much more efficient to let them focus on the scientific questions surrounding data analysis, and reduce the amount of resources they spend on data management and access. This is part of a larger cultural change\u2014which has wide impact on the evolution of data services\u2014and improving findability is a key concern.\n<\/p><p>A key capability necessary to support future scenarios is, therefore, support at the data infrastructure level for better automation of the processes dealing with data and workflows. Out of the many possible facets related to this challenge that could be derived from the FAIR principles, we focus on the automation of findability (principles F1-F3), emphasizing that identifiers are a foundational element from which the other principles must follow.<sup id=\"rdp-ebb-cite_ref-JutyUnique20_7-0\" class=\"reference\"><a href=\"#cite_note-JutyUnique20-7\">[7]<\/a><\/sup> A key question is: How can automated processes help to make more data and workflows findable, particularly from early research workflow stages? In this article, we understand an automated process as one that is capable of limited, autonomous decision-making. This is driven by rule systems specified by humans, but could also, in a later evolution, be replaced by means of <a href=\"https:\/\/www.limswiki.org\/index.php\/Machine_learning\" title=\"Machine learning\" class=\"wiki-link\" data-key=\"79aab39cfa124c958cd1dbcab3dde122\">machine learning<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Essential_requirements_for_automating_data_and_workflow_findability_for_machines\">Essential requirements for automating data and workflow findability for machines<\/span><\/h2>\n<p>To support machine-actionable processes in data infrastructures and VREs, objects (including data and workflows, but possibly also other artifacts) need to be persistently identifiable, independent of location (F1).<sup id=\"rdp-ebb-cite_ref-JutyUnique20_7-1\" class=\"reference\"><a href=\"#cite_note-JutyUnique20-7\">[7]<\/a><\/sup> This is the primary prerequisite for any other benefits to be realized. An important constraint is that the future preservation status of an object is likely unclear at early stages, but identification is nonetheless required. This shapes the choice of persistent identifier (PID) systems to employ.\n<\/p><p>Moreover, offering elemental operations on objects, independent of their location, readily supports the needs of data management processes within data infrastructures. We can define two levels of such operations. The first level includes \u201ccreate, read, update. and delete\u201d (CRUD) operations on single objects and collections, as well as directly related support operations such as retrieving object metadata, which is critical to facilitate findability. The second level consists of more complex operations such as creating a replica (physically copy an object, then update associated metadata to describe the replica) or creating a success or new version (create a new object, the describe the relation with the predecessor in metadata).\n<\/p><p>In order to record a fundamental level of provenance from data processing in VREs, persistent identification must be complemented by additional metadata (F2). Following the W3C provenance (PROV) data model <sup id=\"rdp-ebb-cite_ref-WC3PROV13_8-0\" class=\"reference\"><a href=\"#cite_note-WC3PROV13-8\">[8]<\/a><\/sup>, the first action is to record the link between input and output data using <tt>wasDerivedFrom<\/tt> relations between entities. Extensions may then add links to the processing script or workflow (activity) and links to the executing user (agent). Finally, to enable automated processes to not only use such information (which can be ensured by relying on PROV-compatible encodings) but also discover it in the first place, the methods by which to access it must be well-defined (A1). This does not necessarily mean that the information has to be centrally stored. It could also be federated, but the mechanism has to be straightforward to use by automated processes, and the use of the identifier to access metadata according to A1 is a primary prerequisite.\n<\/p><p>One particular aspect for data processing is that workflows may be defined by users, but those worflows will be best used by automated processes as (web) services, i.e., they must have a machine-interpretable service description and standardized interfaces, and give back unambiguous information about execution results. A mechanism that makes such information readily available for automated agents is required.\n<\/p><p>One well-known established approach for addressing concerns of reproducibility, automation, and provenance, in particular, are scientific workflow systems.<sup id=\"rdp-ebb-cite_ref-TaylorWork07_9-0\" class=\"reference\"><a href=\"#cite_note-TaylorWork07-9\">[9]<\/a><\/sup> These have seen larger adoption in the \u201comics\u201d-based research area, but they are less adopted for climate or geophysics data processing scenarios, in contrast to the adoption of interactive Python via Jupyter Notebooks. One contributing factor may be that interactive notebooks support both repetitive tasks and exploratory work modes. Exploration is, for instance, a major factor for geophysics use cases in the Scientific Software Solution Centre (SSSC)<sup id=\"rdp-ebb-cite_ref-SquireScient18_2-1\" class=\"reference\"><a href=\"#cite_note-SquireScient18-2\">[2]<\/a><\/sup>, where the set of possible input data is relatively small, but there is a large variation in algorithms to evaluate. Cases like this make the Jupyter Notebook particularly attractive. For HPC environments and data infrastructures, integration with fairly heavy-weight workflow systems poses a huge challenge that is often refrained from due to the large investment required. Nonetheless, the controlled environment of a workflow system has a certain appeal for automating metadata capture in the background (F2, R1), yet in view of the tremendous user adoption of interactive notebooks, a solution should potentially be suitable for both approaches.\n<\/p><p>Finally, a few other general requirements will be critical to ensure practical success of any solution. It should be resilient against operational incidents, such as temporary server outages, and either recover from them without human intervention or fully fail over, ensuring continuous operation. While this may seem a generally good requirement for any technical system, it is even more critically so if the system is built on automated processes capable of limited autonomy.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Elements_of_a_possible_solution\">Elements of a possible solution<\/span><\/h2>\n<p>A consistent implementation that improves findability for machines can be broken down into several parts, following the research workflow. Ultimately, findability depends on trustworthy, gapless metadata, and improving the processes in the workflow contributes to better findability at the consumer's end. One important constraint is that even a full solution will still be semi-automatic, i.e., a human user will still need to be involved as the ultimate referee\u2014a contributor of some metadata elements that only a human can define\u2014and to ensure overall <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a>.\n<\/p><p>The persistent identification of objects and reciprocal association (F3) with machine-interpretable metadata can be facilitated by employing the Digital Object Architecture (DOA)<sup id=\"rdp-ebb-cite_ref-KahnAFrame06_10-0\" class=\"reference\"><a href=\"#cite_note-KahnAFrame06-10\">[10]<\/a><\/sup>, PID Kernel Information<sup id=\"rdp-ebb-cite_ref-WeigelRDA18_11-0\" class=\"reference\"><a href=\"#cite_note-WeigelRDA18-11\">[11]<\/a><\/sup>, and Data Type Registries<sup id=\"rdp-ebb-cite_ref-LannomData15_12-0\" class=\"reference\"><a href=\"#cite_note-LannomData15-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SchwardmannAuto16_13-0\" class=\"reference\"><a href=\"#cite_note-SchwardmannAuto16-13\">[13]<\/a><\/sup>, while also following the model of FAIR Digital Objects.<sup id=\"rdp-ebb-cite_ref-ECTurning18_14-0\" class=\"reference\"><a href=\"#cite_note-ECTurning18-14\">[14]<\/a><\/sup>\n<\/p><p>In the following, we describe along a generalized workflow how these elements can be combined to address the requirements. A key aspect is that the PID is seen as the primary anchor or \u201centry point\u201d for any agent interacting with an object, making availability and proper maintenance of PIDs a necessary pre-condition.\n<\/p><p>As a first step, any object should receive a PID. At the earliest workflow stages, descriptive metadata are likely not available yet, and the long-term preservation status of an object is also unclear. Therefore, a PID system that does not mandate specific metadata elements and does not enforce strong policies, such as the Handle System, is a good fit. Even more important than the choice of PID system is that, in order to support lateroperations and autonomous handling of objects, the PID should be embedded in the object. For instance, if the object is a file with a format supporting embedded metadata, the PID should be included.\n<\/p><p>While descriptive metadata may not be available, support for generalized CRUD operations requires essential structural and administrative metadata to be captured, stored, and made available for requestors. Metadata capture must be highly automated and reliable, both in terms of technical reliability and ensured metadata quality. This requires an approach that may be very different from established procedures. For example, in the case of adoption by the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/esgf.llnl.gov\/\" target=\"_blank\">Earth System Grid Federation<\/a> (ESGF) for climate data, it became clear early on that technical solutions must be embedded in processes agreed upon by all stakeholders (users, project and data managers, infrastructure providers, and administrators), and that defining and establishing these processes is a prerequisite for subsequent technical development. This leads to a general observation that, in particular, the quality of metadata may be controlled by technical means, but high quality can only be achieved if the processes are supported by all stakeholders.\n<\/p><p>Metadata delivery must work with low latency and in highly standardized, machine-interpretable encodings. While originally addressed toward encoding provenance, the concept of PID Kernel Information and its underlying principles can fulfill these additional requirements also to support metadata required for CRUD operations. At later workflow stages of data publication and wide sharing, PID Kernel Information is unsuitable to feed search catalogs with meaningful descriptive metadata, a process that must be complemented with other sources. Here, it may be possible to leverage existing workflows driven by the need for descriptive metadata for purposes of archival and credit-giving.\n<\/p><p>Common operations on objects may best be implemented according to a comprehensive specification. In the DOA framework, a Digital Object Interface Protocol (DOIP) has been defined to additionally incorporate such operation specifications. Implementing such a protocol on top of not only a single repository, but also as part of a service-oriented architecture may be more difficult, since it is likely that any single operation is offered by multiple middleware services, and that execution of an operation may have side effects or require actions to be taken by other services. A practical obstacle for implementation is the required compatibility with existing architectural components, protocols, and interfaces (e.g., REST), and fitness for use within distributed systems, where no single control point may exist to coordinate the execution of operations.\n<\/p><p>Services and VREs for data analysis and processing such as the Scientific Software Solution Centre (SSSC)<sup id=\"rdp-ebb-cite_ref-SquireScient18_2-2\" class=\"reference\"><a href=\"#cite_note-SquireScient18-2\">[2]<\/a><\/sup> or the ENES Climate Analytics Service (ECAS)<sup id=\"rdp-ebb-cite_ref-BendoukhaENES18_15-0\" class=\"reference\"><a href=\"#cite_note-BendoukhaENES18-15\">[15]<\/a><\/sup> present a unique opportunity to implement a solution at small scale in a relatively closed environment, since their supported workflows are a smaller but representative subset of the more general research data workflow, and the central control over the VRE workflows makes implementation easier compared to distributed middleware in larger data infrastructures. Implementing automated PID assignment, metadata generation and provenance capture, and elemental object operations in a VRE may easily demonstrate improvements to downstream findability that can inform decisions on implementations in larger infrastructures.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Extending_capabilities_with_machine_learning\">Extending capabilities with machine learning<\/span><\/h2>\n<p>We will briefly highlight two opportunities for a solution to employ machine learning techniques, concerning classification for search catalogs (F4) and building recommender systems. While the components mentioned so far can improve metadata acquisition, it is likely that gaps will remain that also cannot be covered through increased human intervention. Machine learning may help by classifying artifacts based on incomplete information, possibly also using unstructured sources such as log files from executing computing jobs or running processing tools. This may work particularly well if a VRE or scientific workflow management system is used, but may also work well in the back of common HPC jobs.\n<\/p><p>The most important constraint is that the result of such classification by machine learning algorithms will bear an intrinsic uncertainty. It should therefore not be a full alternative to metadata acquisition, particularly in view of the level of precision required for data preservation, but it can fuel search catalogs, as a level of uncertainty may be tolerable. Information both out of metadata and algorithmic classification may then be used to power recommender systems that enhance search catalog capabilities.<sup id=\"rdp-ebb-cite_ref-DevarajuAHybrid18_16-0\" class=\"reference\"><a href=\"#cite_note-DevarajuAHybrid18-16\">[16]<\/a><\/sup> They may recommend, for example, input data sets or workflows for reuse to VRE users, and thus contribute to improved findability from the consumer's end.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Outlook_and_conclusions\">Outlook and conclusions<\/span><\/h2>\n<p>We have touched upon important requirements and key elements of a solution to improve findability of data and workflows by leveraging automation capabilities during the research workflow. Future work on the topic may derive more concrete recommendations and build demonstrators. The approach described, motivated by requirements seen in several disciplines, hints at promising solutions that could emerge out of collaborative work across disciplinary infrastructures and service providers.\n<\/p><p>In the end, a decisive take on automation for findability can be of benefit to multiple stakeholders. Researchers producing data can spend less time on data management and documentation, researchers reusing data and workflows will have access to metadata on a wider range of objects, and research administrators and funders may benefit from deeper insight into the impact of data-generating workflows. The wider adoption of a solution may also benefit other aspects of FAIR, e.g., interoperability and reusability.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>All authors have made meaningful and valuable contributions in revising and proofreading the resulting manuscript. Tobias Weigel has led the editorial process.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-BalajiRequire18-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BalajiRequire18_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Balaji, V.; Taylor, K.E.; Juckes, M. et al. (2018). \"Requirements for a global data infrastructure in support of CMIP6\". <i>Geoscientific Model Development<\/i> <b>11<\/b> (9): 3659\u20133680. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5194%2Fgmd-11-3659-2018\" target=\"_blank\">10.5194\/gmd-11-3659-2018<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Requirements+for+a+global+data+infrastructure+in+support+of+CMIP6&rft.jtitle=Geoscientific+Model+Development&rft.aulast=Balaji%2C+V.%3B+Taylor%2C+K.E.%3B+Juckes%2C+M.+et+al.&rft.au=Balaji%2C+V.%3B+Taylor%2C+K.E.%3B+Juckes%2C+M.+et+al.&rft.date=2018&rft.volume=11&rft.issue=9&rft.pages=3659%E2%80%933680&rft_id=info:doi\/10.5194%2Fgmd-11-3659-2018&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SquireScient18-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SquireScient18_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-SquireScient18_2-1\">2.1<\/a><\/sup> <sup><a href=\"#cite_ref-SquireScient18_2-2\">2.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Squire, G.; Wu, M.; Friedrich, C. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/agu.confex.com\/agu\/fm18\/meetingapp.cgi\/Paper\/459873\" target=\"_blank\">\"IN43C-0903: Scientific Software Solution Centre for Discovering, Sharing and Reusing Research Software\"<\/a>. <i>Proceedings from the 2018 AGU Fall Meeting<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/agu.confex.com\/agu\/fm18\/meetingapp.cgi\/Paper\/459873\" target=\"_blank\">https:\/\/agu.confex.com\/agu\/fm18\/meetingapp.cgi\/Paper\/459873<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=IN43C-0903%3A+Scientific+Software+Solution+Centre+for+Discovering%2C+Sharing+and+Reusing+Research+Software&rft.jtitle=Proceedings+from+the+2018+AGU+Fall+Meeting&rft.aulast=Squire%2C+G.%3B+Wu%2C+M.%3B+Friedrich%2C+C.+et+al.&rft.au=Squire%2C+G.%3B+Wu%2C+M.%3B+Friedrich%2C+C.+et+al.&rft.date=2018&rft_id=https%3A%2F%2Fagu.confex.com%2Fagu%2Ffm18%2Fmeetingapp.cgi%2FPaper%2F459873&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GobleFAIR20-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GobleFAIR20_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goble, C.; Cohen=Boulakia, S.; Soiland-Reyes, S. et al. (2020). \"FAIR Computational Workflows\". <i>Data Intelligence<\/i> <b>2<\/b> (1\u20132): 108\u201321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1162%2Fdint_a_00033\" target=\"_blank\">10.1162\/dint_a_00033<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=FAIR+Computational+Workflows&rft.jtitle=Data+Intelligence&rft.aulast=Goble%2C+C.%3B+Cohen%3DBoulakia%2C+S.%3B+Soiland-Reyes%2C+S.+et+al.&rft.au=Goble%2C+C.%3B+Cohen%3DBoulakia%2C+S.%3B+Soiland-Reyes%2C+S.+et+al.&rft.date=2020&rft.volume=2&rft.issue=1%E2%80%932&rft.pages=108%E2%80%9321&rft_id=info:doi\/10.1162%2Fdint_a_00033&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MonsCloudy17-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MonsCloudy17_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mons, B.; Neylon, C.; Velterop, J. et al. (2017). \"Cloudy, increasingly FAIR; revisiting the FAIR Data guiding principles for the European Open Science Cloud\". <i>Information Services & Use<\/i> <b>37<\/b> (1): 49\u201356. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2FISU-170824\" target=\"_blank\">10.3233\/ISU-170824<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cloudy%2C+increasingly+FAIR%3B+revisiting+the+FAIR+Data+guiding+principles+for+the+European+Open+Science+Cloud&rft.jtitle=Information+Services+%26+Use&rft.aulast=Mons%2C+B.%3B+Neylon%2C+C.%3B+Velterop%2C+J.+et+al.&rft.au=Mons%2C+B.%3B+Neylon%2C+C.%3B+Velterop%2C+J.+et+al.&rft.date=2017&rft.volume=37&rft.issue=1&rft.pages=49%E2%80%9356&rft_id=info:doi\/10.3233%2FISU-170824&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WybornBuild17-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WybornBuild17_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wyborn, L.A.; Fraser, R.; Evans, B.J.K. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/agu.confex.com\/agu\/fm17\/meetingapp.cgi\/Paper\/293857\" target=\"_blank\">\"ED32B-03: Building a Generic Virtual Research Environment Framework for Multiple Earth and Space Science Domains and a Diversity of Users\"<\/a>. <i>Proceedings from the 2017 AGU Fall Meeting<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/agu.confex.com\/agu\/fm17\/meetingapp.cgi\/Paper\/293857\" target=\"_blank\">https:\/\/agu.confex.com\/agu\/fm17\/meetingapp.cgi\/Paper\/293857<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ED32B-03%3A+Building+a+Generic+Virtual+Research+Environment+Framework+for+Multiple+Earth+and+Space+Science+Domains+and+a+Diversity+of+Users&rft.jtitle=Proceedings+from+the+2017+AGU+Fall+Meeting&rft.aulast=Wyborn%2C+L.A.%3B+Fraser%2C+R.%3B+Evans%2C+B.J.K.+et+al.&rft.au=Wyborn%2C+L.A.%3B+Fraser%2C+R.%3B+Evans%2C+B.J.K.+et+al.&rft.date=2017&rft_id=https%3A%2F%2Fagu.confex.com%2Fagu%2Ffm17%2Fmeetingapp.cgi%2FPaper%2F293857&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarkerTheGlob19-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BarkerTheGlob19_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barker, M.; Olabarriaga, S.D.; Wilkins-Diehr, N. et al. (2019). \"The global impact of science gateways, virtual research environments and virtual laboratories\". <i>Future Generation Computer Systems<\/i> <b>95<\/b>: 240\u201348. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.future.2018.12.026\" target=\"_blank\">10.1016\/j.future.2018.12.026<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+global+impact+of+science+gateways%2C+virtual+research+environments+and+virtual+laboratories&rft.jtitle=Future+Generation+Computer+Systems&rft.aulast=Barker%2C+M.%3B+Olabarriaga%2C+S.D.%3B+Wilkins-Diehr%2C+N.+et+al.&rft.au=Barker%2C+M.%3B+Olabarriaga%2C+S.D.%3B+Wilkins-Diehr%2C+N.+et+al.&rft.date=2019&rft.volume=95&rft.pages=240%E2%80%9348&rft_id=info:doi\/10.1016%2Fj.future.2018.12.026&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JutyUnique20-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JutyUnique20_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-JutyUnique20_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Juty, N.; Wimalaratne, S.M.; Soiland-Reyes, S. et al. (2020). \"Unique, Persistent, Resolvable: Identifiers as the Foundation of FAIR\". <i>Data Intelligence<\/i> <b>2<\/b> (1\u20132): 30\u201339. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1162%2Fdint_a_00025\" target=\"_blank\">10.1162\/dint_a_00025<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Unique%2C+Persistent%2C+Resolvable%3A+Identifiers+as+the+Foundation+of+FAIR&rft.jtitle=Data+Intelligence&rft.aulast=Juty%2C+N.%3B+Wimalaratne%2C+S.M.%3B+Soiland-Reyes%2C+S.+et+al.&rft.au=Juty%2C+N.%3B+Wimalaratne%2C+S.M.%3B+Soiland-Reyes%2C+S.+et+al.&rft.date=2020&rft.volume=2&rft.issue=1%E2%80%932&rft.pages=30%E2%80%9339&rft_id=info:doi\/10.1162%2Fdint_a_00025&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WC3PROV13-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WC3PROV13_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Belhajjame, K. B'Far, R.; Cheney, J. et al. (30 April 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.w3.org\/TR\/2013\/REC-prov-dm-20130430\/\" target=\"_blank\">\"PROV-DM: The PROV Data Model\"<\/a>. <i>w3.org<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.w3.org\/TR\/2013\/REC-prov-dm-20130430\/\" target=\"_blank\">https:\/\/www.w3.org\/TR\/2013\/REC-prov-dm-20130430\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=PROV-DM%3A+The+PROV+Data+Model&rft.atitle=w3.org&rft.aulast=Belhajjame%2C+K.+B%27Far%2C+R.%3B+Cheney%2C+J.+et+al.&rft.au=Belhajjame%2C+K.+B%27Far%2C+R.%3B+Cheney%2C+J.+et+al.&rft.date=30+April+2013&rft_id=https%3A%2F%2Fwww.w3.org%2FTR%2F2013%2FREC-prov-dm-20130430%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorWork07-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorWork07_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Taylor, I.J.; Deelman, E.; Gannon, D.B. et al., ed. (2007). <i>Workflows fo e-Science<\/i>. Springer. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781846287572.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Workflows+fo+e-Science&rft.date=2007&rft.pub=Springer&rft.isbn=9781846287572&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KahnAFrame06-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KahnAFrame06_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kahn, R.; Wilensky, R. (2006). \"A framework for distributed digital object services\". <i>International Journal on Digital Libraries<\/i> <b>6<\/b>: 115\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00799-005-0128-x\" target=\"_blank\">10.1007\/s00799-005-0128-x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+framework+for+distributed+digital+object+services&rft.jtitle=International+Journal+on+Digital+Libraries&rft.aulast=Kahn%2C+R.%3B+Wilensky%2C+R.&rft.au=Kahn%2C+R.%3B+Wilensky%2C+R.&rft.date=2006&rft.volume=6&rft.pages=115%E2%80%9323&rft_id=info:doi\/10.1007%2Fs00799-005-0128-x&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeigelRDA18-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WeigelRDA18_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Weigel, T.; Plale, B.; Parsons, M. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.rd-alliance.org\/group\/pid-kernel-information-wg\/outcomes\/recommendation-pid-kernel-information\" target=\"_blank\">\"RDA Recommendation on PID Kernel Information\"<\/a>. <i>Research Data Alliance<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.15497%2Frda00031\" target=\"_blank\">10.15497\/rda00031<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.rd-alliance.org\/group\/pid-kernel-information-wg\/outcomes\/recommendation-pid-kernel-information\" target=\"_blank\">https:\/\/www.rd-alliance.org\/group\/pid-kernel-information-wg\/outcomes\/recommendation-pid-kernel-information<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=RDA+Recommendation+on+PID+Kernel+Information&rft.atitle=Research+Data+Alliance&rft.aulast=Weigel%2C+T.%3B+Plale%2C+B.%3B+Parsons%2C+M.+et+al.&rft.au=Weigel%2C+T.%3B+Plale%2C+B.%3B+Parsons%2C+M.+et+al.&rft.date=2018&rft_id=info:doi\/10.15497%2Frda00031&rft_id=https%3A%2F%2Fwww.rd-alliance.org%2Fgroup%2Fpid-kernel-information-wg%2Foutcomes%2Frecommendation-pid-kernel-information&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LannomData15-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LannomData15_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Lannom, L.; Broeder, D.; Manepalli, G. (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.rd-alliance.org\/group\/data-type-registries-wg\/outcomes\/data-type-registries\" target=\"_blank\">\"Data Type Registries working group output\"<\/a>. <i>Research Data Alliance<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.15497%2FA5BCD108-ECC4-41BE-91A7-20112FF77458\" target=\"_blank\">10.15497\/A5BCD108-ECC4-41BE-91A7-20112FF77458<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.rd-alliance.org\/group\/data-type-registries-wg\/outcomes\/data-type-registries\" target=\"_blank\">https:\/\/www.rd-alliance.org\/group\/data-type-registries-wg\/outcomes\/data-type-registries<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+Type+Registries+working+group+output&rft.atitle=Research+Data+Alliance&rft.aulast=Lannom%2C+L.%3B+Broeder%2C+D.%3B+Manepalli%2C+G.&rft.au=Lannom%2C+L.%3B+Broeder%2C+D.%3B+Manepalli%2C+G.&rft.date=2015&rft_id=info:doi\/10.15497%2FA5BCD108-ECC4-41BE-91A7-20112FF77458&rft_id=https%3A%2F%2Fwww.rd-alliance.org%2Fgroup%2Fdata-type-registries-wg%2Foutcomes%2Fdata-type-registries&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchwardmannAuto16-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchwardmannAuto16_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schwardmann, U. (2016). \"Automated schema extraction for PID information types\". <i>Proceedings of the 2016 IEEE International Conference on Big Data<\/i>: 3036\u20133044. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FBigData.2016.7840957\" target=\"_blank\">10.1109\/BigData.2016.7840957<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automated+schema+extraction+for+PID+information+types&rft.jtitle=Proceedings+of+the+2016+IEEE+International+Conference+on+Big+Data&rft.aulast=Schwardmann%2C+U.&rft.au=Schwardmann%2C+U.&rft.date=2016&rft.pages=3036%E2%80%933044&rft_id=info:doi\/10.1109%2FBigData.2016.7840957&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ECTurning18-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ECTurning18_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">European Commission (2018). <i>Turning FAIR Into Reality<\/i>. European Commission. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2777%2F1524\" target=\"_blank\">10.2777\/1524<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9789279965463.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Turning+FAIR+Into+Reality&rft.aulast=European+Commission&rft.au=European+Commission&rft.date=2018&rft.pub=European+Commission&rft_id=info:doi\/10.2777%2F1524&rft.isbn=9789279965463&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BendoukhaENES18-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BendoukhaENES18_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bendoukha, S.; Weigel, T.; Fiore, S. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/meetingorganizer.copernicus.org\/EGU2018\/EGU2018-12549.pdf\" target=\"_blank\">\"ENES Climate Analytics Service (ECAS)\"<\/a> (PDF). <i>Geophysical Researh Abstracts<\/i> <b>20<\/b> (EGU2018-12549)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/meetingorganizer.copernicus.org\/EGU2018\/EGU2018-12549.pdf\" target=\"_blank\">https:\/\/meetingorganizer.copernicus.org\/EGU2018\/EGU2018-12549.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ENES+Climate+Analytics+Service+%28ECAS%29&rft.jtitle=Geophysical+Researh+Abstracts&rft.aulast=Bendoukha%2C+S.%3B+Weigel%2C+T.%3B+Fiore%2C+S.+et+al.&rft.au=Bendoukha%2C+S.%3B+Weigel%2C+T.%3B+Fiore%2C+S.+et+al.&rft.date=2018&rft.volume=20&rft.issue=EGU2018-12549&rft_id=https%3A%2F%2Fmeetingorganizer.copernicus.org%2FEGU2018%2FEGU2018-12549.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DevarajuAHybrid18-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DevarajuAHybrid18_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Devaraju, A.; Berkovsky, S. (2018). \"A Hybrid Recommendation Approach for Open Research Datasets\". <i>Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization<\/i>: 207\u201311. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3209219.3209250\" target=\"_blank\">10.1145\/3209219.3209250<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Hybrid+Recommendation+Approach+for+Open+Research+Datasets&rft.jtitle=Proceedings+of+the+26th+Conference+on+User+Modeling%2C+Adaptation+and+Personalization&rft.aulast=Devaraju%2C+A.%3B+Berkovsky%2C+S.&rft.au=Devaraju%2C+A.%3B+Berkovsky%2C+S.&rft.date=2018&rft.pages=207%E2%80%9311&rft_id=info:doi\/10.1145%2F3209219.3209250&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_data_and_workflows_findable_for_machines\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194108\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.273 seconds\nReal time usage: 0.291 seconds\nPreprocessor visited node count: 12958\/1000000\nPreprocessor generated node count: 33889\/1000000\nPost\u2010expand include size: 84262\/2097152 bytes\nTemplate argument size: 30592\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 269.808 1 - -total\n 80.53% 217.264 1 - Template:Reflist\n 66.44% 179.250 16 - Template:Citation\/core\n 49.01% 132.241 11 - Template:Cite_journal\n 13.91% 37.541 1 - Template:Infobox_journal_article\n 13.31% 35.898 1 - Template:Infobox\n 12.79% 34.503 3 - Template:Cite_web\n 9.17% 24.729 2 - Template:Cite_book\n 9.02% 24.338 80 - Template:Infobox\/row\n 5.24% 14.129 13 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12312-0!*!0!!en!*!* and timestamp 20210429194107 and revision id 41470\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Making_data_and_workflows_findable_for_machines\">https:\/\/www.limswiki.org\/index.php\/Journal:Making_data_and_workflows_findable_for_machines<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","79a0829b6677ab7c03477c2b7c49428a_images":[],"79a0829b6677ab7c03477c2b7c49428a_timestamp":1619725267,"784b63abe742a2cc6286e50d625220ec_type":"article","784b63abe742a2cc6286e50d625220ec_title":"Named data networking for genomics data management and integrated workflows (Ogle et al. 2021)","784b63abe742a2cc6286e50d625220ec_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows","784b63abe742a2cc6286e50d625220ec_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Named data networking for genomics data management and integrated workflows\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nNamed data networking for genomics data management and integrated workflowsJournal\n \nFrontiers in Big DataAuthor(s)\n \nOgle, Cameron; Reddick, David; McKnight, Coleman; Biggs, Tyler; Pauly, Rini; Ficklin, Stephen P.; Feltus, F. Alex; Shannigrahi, SusmitAuthor affiliation(s)\n \nClemson University, Tennessee Tech University, Washington State UniversityPrimary contact\n \nEmail: sshannigrahi at tntech dot eduEditors\n \nShi, F.Year published\n \n2021Volume and issue\n \n4Article #\n \n582468DOI\n \n10.3389\/fdata.2021.582468ISSN\n \n2624-909XDistribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2021.582468\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2021.582468\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Named data networking \n\n3.1 Hierarchical naming \n3.2 Data-centric security \n3.3 In-network caching \n\n\n4 Genomics cyberinfrastructure challenges and solutions using NDN \n\n4.1 The massive storage problem \n\n4.1.1 NDN-based solution \n\n\n4.2 The data discovery problem \n\n4.2.1 NDN-based solution \n\n\n4.3 The fast and scalable data access problem \n\n4.3.1 NDN-based solution \n\n\n4.4 The problem of minimizing transfer volume \n\n4.4.1 NDN-based solution \n\n\n4.5 The secure collaboration problem \n\n4.5.1 NDN-based solution \n\n\n\n\n5 Method \n\n5.1 NDN testbed \n5.2 Data naming and publication \n5.3 Integration with GEMmaker \n\n\n6 Results \n\n6.1 Performance evaluation \n\n\n7 Codebase \n8 Discussion and future directions \n\n8.1 Discussion \n\n8.1.1 Economic considerations \n\n\n8.2 Future directions \n\n8.2.1 Software performance \n8.2.2 Accessing distributed data over NDN= \n8.2.3 Distributed repositories over NDN \n8.2.4 Publication of more genomics datasets and metadata into the NDN testbed \n8.2.5 Integration with Docker \n8.2.6 Integration With Kubernetes \n\n\n8.3 Limitations \n\n\n9 Conclusion \n10 Acknowledgements \n\n10.1 Author contributions \n10.2 Data availability statement \n10.3 Funding \n10.4 Conflict of interest \n\n\n11 References \n12 Notes \n\n\n\nAbstract \nAdvanced imaging and DNA sequencing technologies now enable the diverse biology community to routinely generate and analyze terabytes of high-resolution biological data. The community is rapidly heading toward the petascale in single-investigator laboratory settings. As evidence, the National Center for Biotechnology Information (NCBI) Sequence Read Archive (SRA) central DNA sequence repository alone contains over 45 petabytes of biological data. Given the geometric growth of this and other genomics repositories, an exabyte of mineable biological data is imminent. The challenges of effectively utilizing these datasets are enormous, as they are not only large in size but also stored in various geographically distributed repositories such as those hosted by the NCBI, as well as in the DNA Data Bank of Japan (DDBJ), European Bioinformatics Institute (EBI), and NASA\u2019s GeneLab. \nIn this work, we first systematically point out the data management challenges of the genomics community. We then introduce named data networking (NDN), a novel but well-researched internet architecture capable of solving these challenges at the network layer. NDN performs all operations such as forwarding requests to data sources, content discovery, access, and retrieval using content names (that are similar to traditional filenames or filepaths), all while eliminating the need for a location layer (the IP address) for data management. Utilizing NDN for genomics workflows simplifies data discovery, speeds up data retrieval using in-network caching of popular datasets, and allows the community to create infrastructure that supports operations such as creating federation of content repositories, retrieval from multiple sources, remote data subsetting, and others. Using name-based operations also streamlines deployment and integration of workflows with various cloud platforms. \nWe make four signigicant contributions with this work. First, we enumerate the cyberinfrastructure challenges of the genomics community that NDN can alleviate. Second, we describe our efforts in applying NDN for a contemporary genomics workflow (GEMmaker) and quantify the improvements. The preliminary evaluation shows a sixfold speed up in data insertion into the workflow. Third, as a pilot, we have used an NDN naming scheme (agreed upon by the community and discussed in the \"Method\" section) to publish data from broadly used data repositories, including the NCBI SRA. We have loaded the NDN testbed with these pre-processed genomes that can be accessed over NDN and used by anyone interested in those datasets. Finally, we discuss our continued effort in integrating NDN with cloud computing platforms, such as the Pacific Research Platform (PRP). \nThe reader should note that the goal of this paper is to introduce NDN to the genomics community and discuss NDN\u2019s properties that can benefit the genomics community. We do not present an extensive performance evaluation of NDN; we are working on extending and evaluating our pilot deployment and will present systematic results in a future work.\nKeywords: genomics data, genomics workflows, large science data, cloud computing, named data networking\n\nIntroduction \nScientific communities are entering a new era of exploration and discovery in many fields, driven by high-density data accumulation. A few examples are climate science[1], high-energy particle physics (HEP)[2], astrophysics[3][4], genomics[5], seismology[6], and biomedical research[7], just to name a few. Often referred to as \u201cdata-intensive\u201d science, these communities utilize and generate extremely large volumes of data, often reaching into the petabytes[8] and soon projected to reach into the exabytes.\nData-intensive science has created radically new opportunities. Take for example high-throughput DNA sequencing (HTDS). Until recently, HTDS was slow and expensive, and only a few institutes were capable of performing it at scale.[9] With the advances in supercomputers, specialized DNA sequencers, and better bioinformatics algorithms, the effectiveness and cost of sequencing has dropped considerably and continues to drop. For example, sequencing the first reference human genome cost around $2.7 billion over 15 years, while currently it costs under $1,000 to resequence a human genome.[10] With commercial incentives, several companies are offering fragmented genome re-sequencing under $100, performed in only a few days. This massive drop in cost and improvement in speed supports more advanced scientific discovery. For example, earlier scientists could only test their hypothesis on a small number of genomes or gene expression conditions within or between species. With more publicly available datasets[5], scientists can test their hypotheses against a larger number of genomes, potentially enabling them to identify rare mutations, precisely classify diseases based on a specific patient, and, thusly, more accurately treat the disease.[11]\nWhile the growth of DNA sequencing is encouraging, it has also created difficulty in genomics data management. For example, the National Center for Biotechnology Information\u2019s (NCBI) Sequence Read Archive (SRA) database hosts 42 petabytes of publicly accessible DNA sequence data.[12] Scientists desiring to use public data must discover (or locate) the data and move it from globally distributed sites to on-premize clusters and distributed computing platforms, including public and commercial clouds. Public repositories such as the NCBI SRA contain a subset of all available genomics data.[13] Similar repositories are hosted by NASA, the National Institutes of Health (NIH), and other organizations. Even though these datasets are highly curated, each public repository uses their own standards for data naming, retrieval, and discovery that makes locating and utilizing these datasets difficult.\nMoreover, data management problems require the community to build and the scientists to spend time learning complex infrastructures (e.g., cloud platforms, grids) and creating tools, scripts, and workflows that can (semi-) automate their research. The current trend of moving from localized institutional storage and computing to an on-demand cloud computing model adds another layer of complexity to the workflows. The next generation of scientific breakthroughs may require massive data. Our ability to manage, distribute, and utilize these types of extreme-scale datasets and securely integrate them with computational platforms may dictate our success (or failure) in future scientific research.\nOur experience in designing and deploying protocols for \"big data\" science[8][14][15][16][17][18] suggests that using hierarchical and community-developed names for storing, discovering, and accessing data can dramatically simplify scientific data management systems (SDMSs), and that the network is the ideal place for integrating domain workflows with distributed services. In this work, we propose a named ecosystem over an evolving but well-researched future internet architecture: named data networking (NDN). NDN utilizes content names for all data management operations such as content addressing, content discovery, and retrieval. Utilizing content names for all network operations massively simplifies data management infrastructure. Users simply ask for the content by name (e.g., \u201c\/ncbi\/homo\/sapiens\/hg38\u201d) and the network delivers the content to the user.\nUsing content names that are understood by the end-user over an NDN network provides multiple advantages: natural caching of popular content near the users, unified access mechanisms, and location-agnostic publication of data and services. For example, a dataset properly named can be downloaded by, for example, NCBI or GeneLab at NASA, whichever is closer to the researcher. Additionally, the derived data (results, annotations, publications) are easily publishable into the network (possibly after vetting and quality control by NCBI or NASA) and immediately discoverable if appropriate naming conventions are agreed upon and followed. Finally, NDN shifts the trust to content itself; each piece of content is cryptographically signed by the data producer and verifiable by anyone for provenance.\nIn this work, we first introduce NDN and the architectural constructs that make it attractive for the genomics community. We then discuss the data management and cyberinfrastructure challenges faced by the genomics community and how NDN can help alleviate them. We then present our pilot study applying NDN to a contemporary genomics workflow GEMmaker[19] and evaluate the integration. Finally, we discuss future research directions and an integration roadmap with cloud computing services.\n\nNamed data networking \nNDN[20] is a new networking paradigm that adopts a drastically different communication model than that current IP model. In NDN, data is accessed by content names (e.g., \u201c\/Human\/DNA\/Genome\/hg38\u201d) rather than through the host where it resides (e.g., ftp:\/\/ftp.ncbi.nlm.nih.gov\/refseq\/H_sapiens\/annotation\/GRCh38_latest\/refseq_identifiers\/GRCh38_latest_genomic.fna.gz). Naming the data allows the network to participate in operations that were not feasible before. Specifically, the network can take part in discovering and local caching of the data, merging similar requests, retrieval from multiple distributed data sources, and more. In NDN, the communication primitive is straightforward (Figure 1): the consumer asks for the content by content name (an \u201cInterest\u201d in NDN terminology), and the network forwards the request toward the publisher.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 1. NDN Forwarding. The two servers on the right announce a namespace (\/google) for the data they serve. The routers make a note of this incoming announcement. When the laptops ask for \/google\/index.html, the routers forward the requests on the appropriate interfaces (31, 32, or both, depending on configuration). Data follows the reverse path.[20]\n\n\n\nFor communication, NDN uses two types of packets: Interest and Data. The content consumer initiates communication in NDN. To retrieve data, a consumer sends out an Interest packet into the network, which carries a name that identifies the desired data. One such content name (similar to a resource identifier) might be \u201c\/google\/index.html\u201d. A network router maintains a name-based forwarding table (FIB) (Figure 1). The router remembers the interface from which the request arrives and then forwards the Interest packet by looking up the name in its FIB. FIBs are populated using a name-based routing protocol such as Named-data Link State Routing Protocol (NLSR).[21]\nNDN routes and forwards packets based on content names[22], which eliminates various problems that addresses pose in the IP architecture, such as address space exhaustion, Network Address Translation (NAT) traversal, mobility, and address management. In NDN, routers perform component-based longest prefix match of the Interest name the FIB. Routing in NDN is similar to IP routing. Instead of announcing IP prefixes, an NDN router announces name prefixes that it is willing to serve (e.g., \u201c\/google\u201d). The announcement is propagated through the network and eventually populates the FIB of every router. Routers match incoming Interests against the FIB using longest prefix match. For example, \u201c\/google\/videos\/movie1.mpg\u201d might match \u201c\/google\u201d or \u201c\/google\/video\u201d. Though an unbounded namespace raises the question of how to maintain control over the routing table sizes and whether looking up variable-length, hierarchical names can be done at line rate, previous works have shown that it is indeed possible to forward packets at 100 Gbps or more.[23][24]\nWhen the Interest reaches a node or router with the requested data, it packages the content under the same name (i.e., the request name), signs it with the producer\u2019s signature, and returns it. For example, a request for \u201c\/google\/index.html\u201d brings back data under the same name \u201c\/google\/index.html\u201d that contains a payload with the actual data and the data producer\u2019s (i.e., Google) signature. This Data packet follows the reverse path taken by the Interest. Note that Interest or Data packets do not carry any host information or IP addresses; they are simply forwarded based on names (for Interest packets) or state in the routers (for Data packets). Since every NDN Data packet is signed, the router can store it locally in a cache to satisfy future requests.\n\nHierarchical naming \nThere is no restriction on how content is named in NDN except they must be human-readable and hierarchical, and globally unique. The scientific communities develop the naming schemes as they see fit, and the uniqueness of names can be ensured by name registrars (similar to existing DNS Registrars).\nThe NDN design assumes hierarchically structured names, e.g., a genome sequence published by NCBI may have the name \u201c\/NCBI\/Human\/DNA\/Genome\/hg38\u201d, where \u201c\/\u201d indicates a separator between name components. The whole sequence may not fit in a single Data packet, so the segments (or chunks) of the sequence will have the names \u201c\/NCBI\/Human\/DNA\/Genome\/hg38\/{1..n}\u201c. Data that is routed and retrieved globally must have a globally unique name. This is achieved by creating a hierarchy of naming components, just like Domain Name System (DNS). In the example above, all sequences under NCBI will potentially reside under \u201c\/NCBI\u201d; \u201c\/NCBI\u201d is the name prefix that will be announced into the network. This hierarchical structure of names is useful both for applications and the network. For applications, it provides an opportunity to create structured, organized names. On the other hand, the network does not need to know all the possible content names, only a prefix. For example, \u201c\/NCBI\u201d is sufficient for forwarding.\n\nData-centric security \nIn NDN, security is built into the content. Each piece of data is signed by the data producer and is carried with the content. Data signatures are mandatory; on receiving the data, applications can decide if they trust the publisher or not. The signature, coupled with data publisher information, enables the determination of data provenance. NDN\u2019s data-centric security helps establish data provenance, e.g., users can verify content with names that begin with \u201c\/NCBI\u201d is digitally signed by NCBI\u2019s key.\nNDN\u2019s data-centric security decouples content from its original publisher and enables in-network caching; it is no longer critical where the data comes from since the client can verify the authenticity of the data. Unsigned data is rejected either in the network or at the receiving client. The receiver can get content from anyone, such as a repository, a router cache, or a neighbor\u2014as well as the original publisher\u2014and verify that the data is authentic.\n\nIn-network caching \nAutomatic in-network caching is enabled by naming data because a router can cache data packets in its content store to satisfy future requests. Unlike today\u2019s Internet, NDN routers can reuse the cached data packets since they have persistent names and the producer\u2019s signature. The cache (or Content Store) is an in-memory buffer that keeps packets temporarily for future requests. Data such as reference genomes can benefit from caching since caching the content near the user speeds up content delivery and reduces the load on the data servers. In addition to the CS, NDN supports persistent, disk-based repositories (repos).[25] These storage devices can support caching at a larger scale and CDN-like functionality without additional application-layer engineering.\nIn our previous work with the climate science and high-energy physics communities, we saw that even though scientific data is large, a strong locality of reference exists. We found that for climate data, even a 1 GB cache in the network speeds up data distribution significantly.[17] We observe similar patterns in the genomics community, where some of the reference genomes are very popular. These caches do not have to be at the core of the network. We anticipate most of the benefits will come from caching at the edge. For example, a large cache provisioned at the network gateway of a lab will benefit the scientists at that lab. In this case, the lab will provision and maintain their caches. If data is popular across many organizations, it is in the operators best interest to cache the data at the core since this will reduce latency and network traffic. Given that the price of data storage has gone down significantly (an 8 TB (8000 GB) hard-drive costs around $150, at the time of writing this paper), it does not significantly add to the operating costs of the labs. Additionally, new routers and switches are increasingly being shipped with storage, reducing the need for additional capital expenditure. Additionally, caching and cache maintenance is automated in NDN (it follows content popularity), eliminating the need to configure and maintain such storage.\nHaving introduced NDN in this section, we now enumerate the genomics data management problems and how NDN can solve them in the following section.\n\nGenomics cyberinfrastructure challenges and solutions using NDN \nThe genomics community has made significant progress in recent decades. However, this progress has not been without challenges. A core challenge, like many other science domains, is data volume. Due to the low-cost sequencing instruments, the genomics community is rapidly approaching petascale data production at sequencing facilities housed in universities, research, and commercial centers. For example, the SRA repository at NCBI in Maryland, United States contains over 45 petabytes of high-throughput DNA sequence data, and there are other similar genomic data repositories around the world.[26][27] These data are complemented with metadata (though not always present or complete) representing evolutionary relationships, biological sample sources, measurement techniques, and biological conditions.[12]\nFurthermore, while a large amount of data is accessible from large repositories such as the NCBI repository, a significant amount of genomics data resides in thousands of institutional repositories.[28][29][30] The current (preferred) way to publish data is to upload it to a central repository, e.g., NCBI, which is time-consuming and often requires effort from the scientists. The massively distributed nature of the data makes the genomics community unique. In other scientific communities, such as high-energy physics (HEP), climate, and astronomy, only a few large scale repositories serve most of the data.[31] For example, the Large Hadron Collider (LHC) produces most of the data for the HEP community at CERN, the telescopes (such as LSST and the to-be-built SKA) produces most of the data for astrophysics, and the supercomputers at various national labs produce climate simulation outputs.[32]\nModern genomic data comes in the form of reference genomes with coordinate-based annotation files, \u201cdynamic\u201d measurements of genome output (e.g., RNA-seq, CHIP-seq), and individual genome resequencing data. Reference genomes are used by many researchers across the world who can benefit from efficient data delivery mechanisms. The dynamic functional genomics and resequencing genomics datasets are often larger in size and of more focused use. All data is typically retrieved using various contemporary technologies such as sneakernet[33], SCP\/FTP, IBM Aspera[34], Globus[35], and iRODS.[36][37] While reference data is often easier to locate and download, the dynamic and resequencing datasets often are not since they are strewn over geographically distributed institutional repositories. \nLocating and retrieving data are not the only problems that the genomics community face. The rest of this section enumerates the cyberinfrastructure requirements of the genomics community, problems encountered due to the current point-to-point TCP\/IP-based model of the internet, and how NDN can solve these.\n\nThe massive storage problem \nThe genomics community is producing more data than it is currently feasible to store locally.[13] This phenomenon will accelerate as modern field-based or hand-held sequencers become more prevalent in individual research labs and commercial sequencing providers. Increasingly, valuable data is at risk of being lost, potentially forever. While the community must invest in storage capacity, the existing storage strategies need to be optimized, such as deduplication of popular datasets (e.g., the reference genomes). Moreover, popular datasets that are often reused must be available quickly and reliably to reduce the need for copying data.\n\nNDN-based solution \nWith NDN, data can come from anywhere, including in-network caches. Fast access to popular data reduces the need to download and store datasets locally. They can be quickly downloaded and deleted after the experiments. Multiple researchers in the same area can benefit from this approach since they no longer need to individually download datasets from NCBI, rather from a in-network cache that is automatically populated by the network. Further, the data downloaded from this cache can be verified publicly for provenance. Another solution is to push the computation to the data. This can be accomplished by adding a lambda (computational function) to the Interest name. The data source (e.g., a data producer or a dedicated service) interprets the lambda upon receiving the Interest and returns the computed results. Scientists don\u2019t have to download and store large datasets every time they need to run an experiment.\n\nThe data discovery problem \nGenomics data is currently published from central locations (e.g., NCBI, NASA). The challenges in data discovery come not only from the fact that one needs to know all the locations of these datasets but also to navigate different naming schemes and discovery mechanisms provided by the hosting entity. There are many community-supported efforts to define controlled vocabularies and ontologies to help describe data.[38][39][40] These metadata then can be parsed, indexed, and organized for data discovery. A scientist, for example, can associate appropriate metadata with source data, resulting data, and data collections. Moreover, the application of metadata to data is non-uniform, non-standard, and often inconsistent, making them difficult to utilize for consistent naming or data discovery.\n\nNDN-based solution \nNDN does not provide data discovery by itself. Once data is named consistently by a certain community or subcommunity, these names can be indexed by a separate application (see our previous work[15] that provides name discovery functions and operated over NDN). Since name discovery in NDN is sufficient for data retrieval\u2014an application can request for this name\u2014no additional steps are necessary. Note that NDN only requires a hierarchical naming structure; how individual communities name their datasets (\/biology\/genome vs. \/genome\/biology) is up to them.[41]\nA distributed catalog[15] that stores the content names is sufficient to provide efficient name discovery. Since an NDN-based catalog will only hold a community-specific set of names (not the actual data), the synchronization, update, and delete operations are lightweight.[42] These names in these catalogs can be added, updated, and deleted as necessary. We refer the reader to our previous work for the details of how such a catalog can be created and maintained in NDN.[15]\n\nThe fast and scalable data access problem \nCurrently, genomics data retrievals range from downloading a significant amount of data from a central data archive (e.g., NCBI) to downloading only the desired data that can be staged on local or cloud storage systems. For example, the researchers often need to retrieve genome reference data on demand for comparison. Downloading large amounts of datasets over long-distance internet links can be slow and error-prone. Further, the current internet model does not work very well over long distance links.[43] Even with extremely high-speed links, it is particularly difficult to utilize all the available bandwidth.\n\nNDN-based solution \nNDN provides access to data from \u201canywhere,\u201d including storage nodes, in-network caches, and any entities that might have the data. This property allows scientists to reuse already downloaded datasets that are nearby (e.g., dataset downloaded by another scientist in the same lab). Additionally, in NDN data follows the content popularity, as it is cached in the in-network devices automatically. The more popular content is, the higher the likelihood it would be cached nearby. All data is digitally signed, ensuring provenance is preserved.\nGetting content fast and from nearby locations may be convenient to download data when needed and delete them when the computation is finished. For example, the reference human genome has been downloaded by us and our students hundreds of times in the last two decades. Secure and verifiable data downloaded on-demand will reduce the amount of storage needed.\n\nThe problem of minimizing transfer volume \nThe massive data volume needed by genomic workflows can easily saturate institutional networks. For example, the size of sequence data (in FASTQ format) or processed sequence alignments (in binary sequence alignment map or BAM format) for one experiment can easily aggregate into terabytes. If stored in online repositories, these data might be downloaded many times by researchers extending existing studies, leading to high bandwidth usage. One solution is to subset the data and download only the necessary portion. However, several challenges remain. For example, if multiple copies of the file exist, the network\/application layers can not take advantage of that to pull different subsets in parallel.\nHowever, depending on the size and type of the analysis being performed, subsetting of the data may not be appropriate. Currently, that means the scientist would be required to download all datasets (or staged at a remote site) before the computation can begin. However, instead of downloading large amounts of data, pushing computation to data might be much more lightweight.\nFor example, to determine if a scientific avenue (e.g., a large-scale experiment with millions of genomes) or dataset is worth pursuing, the scientists often run smaller-scale experiments for early signs of interesting properties. A key issue is determining the smallest number of records required to produce the same scientific result as the full dataset, and we previously point to a simple saturation point as determined by transcript detection.[44] Once a saturation point has been reached, one could pause and examine the results. If there is an interesting signal, then there is nothing preventing the user from processing more sequence records. However, if there is no signal, one could drop the experiment and move on to other datasets. However, this method currently requires downloading the full experimental datasets and running computations against them.\n\nNDN-based solution \nNDN supports subsetting the data at the source, and transferring only the necessary portions reduces bandwidth consumption. This is already possible through BAM file slicing to select data specific to genomic regions. With NDN, the request can carry the required subsetting parameters and allow the user\/applications to download only the part of the data required for computation. NDN can also parallelize subsetting in the event that multiple slices are needed, and data is replicated over multiple repositories. When subsetting is not appropriate, NDN is able to push computation to data by appending the computation to the Interest name (or adding them as the payload to the interest). The result comes back to the requester under the same name and is also cached for future use, reducing bandwidth usage. Furthermore, in some genomics workflows, caching of computation can reduce the load on the compute and servers (such as those hosted in NCBI or cloud platforms).\n\nThe secure collaboration problem \nGenomics data, especially unpublished or identifiable human data, can be very sensitive. Scientists often need to secure data due to privacy requirements, non-disclosure agreements, or legal restrictions (e.g., HIPPA). Without a security framework, securing data and enforcing permissions becomes difficult. Suitable data access methods with proper access control is therefore required for privacy and legal requirements. At the same time, scientific collaborations often need to share data between groups without violating security restrictions. Suitable frameworks must exist for utilizing open-source sequenced data for research, albeit with appropriately restricted access. The lack of an infrastructure that allows secure access to a large number of sequenced human genomes prevents population genetics researchers from identifying rare mutations or test hypotheses on analogous experiments which can lead to medical advancement. Encryption and data security models, along with proper access control is highly necessary as data breaches of protected data can lead to massive fines, forcing institutions to severely limit the scope of allowable controlled data access on local cyberinfrastructure.\n\nNDN-based solution \nTo support secure data sharing among collaborators, all data in NDN is digitally signed, providing data provenance. When privacy is needed, NDN allows encryption of content, facilitating secure collaborations. Furthermore, the verification of trust and associated operations (such as decryption) can be automated in NDN; this is called schematized trust.[45] One example of schematized trust might be the following: a scientist attempting to decrypt a data packet starting with \u201c\/NCBI\u201d must also present a key that is signed by \u201c\/NCBI\u201d and begins with the \u201c\/NCBI\/scientistA\u201d. More complex, name-based trust schemes are also possible.\nThis section discussed NDN properties that can address data management and cyberinfrastructure challenges faced by the genomics community. In the following section, we present a pilot study that uses a current genomics workflow that demonstrates some of these improvements in a real-world scenario.\n\nMethod \nTo demonstrate how NDN can benefit genomics workflows, we integrated NDN with a current genomics workflow (GEMmaker) and deployed our integrated solution over a real, distributed NDN testbed.[16] The experiment has multiple parts: 1) naming data in a way that is understood by NDN as well as acceptable to the genomics community (Figure 2); 2) publishing data into the testbed and making them discoverable to the users using a distributed catalog and a UI (Figure 3); 3) modifying GEMmaker to interact with the data published in the testbed; and 4) comparing the performance of the new integration to the existing workflow. The following sections describe these efforts in detail.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 2. Genome naming strategy for indexing in NDN. The NDN names were translated from the existing Pynome file naming scheme. Tokens above (surrounded by square brackets) indicate the location of taxonomic names and genome assembly names. Most of the names directly map to hierarchical NDN names. Depending on the use case, components can be added or removed. These names are starting points for all NDN based operations. They are also the only necessary component for an NDN network.\n\n\n\n\n\n\n\n\n\n\n\n\nFigure 3. The user interface (UI) for the NDN-based genomics catalog system. It demonstrates how a name based catalog can act as a central point to an NDN ecosystem. Once the user looks up the names, additional functionality such as data retrieval can be built on top these names.\n\n\n\nNDN testbed \nFor this work, we utilized a geographically distributed six-node testbed that was deployed at Colorado State University, ESNet, and UCAR supercomputing center. The testbed had six high-performance nodes (each with 40 cores, 128 GB memory, and 50 TB storage) and connected over ESnet[46] using 10 Gbps dedicated links. All nodes ran the latest version of Fedora, and the network stack was tuned for big-data transfers.[43] Specifically, we tuned the network interfaces to increase the buffer size and used large ethernet frames (9000-byte jumbo frames). We also tuned the TCP stack according to the ESnet specification, including increasing read and write buffers, utilizing cubic and htcp congestion control algorithms. We also tuned the UDP stack to increase read\/write buffers as well as specifying CPU cores.[43]\n\nData naming and publication \nNDN recommends globally unique, hierarchical, and semantically meaningful names. This is a natural fit for the genomics community since they have established taxonomies dating as far back as 1773.\nNote that while NDN requires names to be globally unique, there is no need for a global convention even among a particular community. For example, two names pointing to the same content\u2014 \/Biology\/Genome\/Homo\/Sapiens and \/NCBI\/Genome\/Homo\/Sapiens\u2014are perfectly acceptable. Each community is free to name their own content as they see fit. The uniqueness in these names come from the first component of the name (the prefix), which is \/Biology and \/NCBI, respectively. We anticipate each entity (e.g., an organization such as NCBI, a university, a community genome project) will have their own namespaces, possibly procured from commercial name registrars, the same way DNS namespaces are obtained today.\nIn the genomics community, very commonly used datasets can even be assigned their own namespaces. For example, a globally unique namespace (e.g., \/human\/genome) may be reserved for human genomes for convenience. However, that special namespace does not preclude an organization from publishing the same genomes from another namespace (e.g., \/NCBI\/human\/genome). While NDN operates on names, it does not interpret the semantic meaning of the name components at the network layer. For example, NCBI announces the \/NCBI prefix into the network that the NDN routers store. When an Interest named \/NCBI\/Genome\/Homo\/Sapiens arrives at the router, the router performs a longest prefix match on the name and matches the interface corresponding to \/NCBI. This way, the network is able to forward Interests and data but does not need to interpret the individual components.\nIt is true that a name component (e.g., \u201chg38\u201d) might have different meaning in different communities. It is the job of the application layer to interpret and use this components as they see fit. In our previous work, we built a catalog (an application) that mapped individual components to their semantic meaning (see Figure 3). Different communities will build different applications on top of NDN to understand the semantic meaning of a name.\nThe other problem is name assignment and reconciliation. In today\u2019s internet, ICCAN[47] and the domain registrars play a significant role in assigning, maintaining and reconciling DNS namespaces (e.g., assigning google.com namespace to Google). In NDN, these organizations will continue to control and assign namespaces that are used over the internet.\nIn NDN, the data producer is responsible for publishing content under a name prefix (\/biology or \/NCBI). NCBI will acquire such namespaces from a name registrar. At that point, only NCBI is allowed to announce the \/NCBI prefix into the internet and publish content under that name prefix. The onus of updating namespaces is on the data publisher and the name registrars. Note that no such NDN name registrar exists today, but we expect similar organizations to exist in the future.\nWhen a user looks up a content name (e.g., \/NCBI\/Genome\/Homo\/Sapiens) in a catalog and expresses an Interest, the Interest is forwarded by the network and eventually reaches the NCBI server that announced \/NCBI. Namespace reconciliation is not necessary for the network to function properly. Let\u2019s imagine NCBI is authorized to publish datasets under two namespaces\u2014\/Biology and \/NCBI\u2014and it publishes the human genome under \u201c\/Biology\/Genome\/Homo\/Sapiens\u201d and \u201c\/NCBI\/Genome\/Homo\/Sapiens\u201d. A user can utilize both names to retrieve content\u2014the network does not need to interpret the meanings of the names\u2014and the interpretation of name and content is up to the applications (and users) requesting and serving the datasets.\nAs part of the NSF SciDAS project[48], we store large amounts of whole genome reference and auxiliary data in a distributed data grid (iRODS). These whole genome references were initially retrieved from the Ensembl database[49] using the Python-based Pynome[50] package and consist of hundreds of Eukaryotic species. Pynome processes these data to provide indexed files which are not available on Ensembl and which are needed for common genomic applications performed by researchers around the world. These data are organized in an evolution-based, hierarchical manner which is an excellent naming convention for an NDN framework.\nFor this study, we used an NDN name translator that we created as part of our previous work[51] to translate these existing names into NDN compatible names. Once translated, the content names became the unique reference for these datasets. Figure 2 shows the reference genome DNA sequence names created from the existing hierarchical naming convention. For example, one such name would look like \u201c\/Absidia\/glauca\/AG_v1\/fa\u201d. We could find that some of these names may or may not contain certain components, for example, infraspecific name in the above example. We then translated and imported these names into our NDN data management framework.[8] All subsequent operations such as discovery, retrieval, and integration with workflows used these names.\nAfter naming, we published these datasets under the derived names on the NDN testbed servers. In this pilot test, we used three nodes to publish the data. However, each testbed server published the same data under the same name, replicating the content. Depending on the client location, requests were routed to the closest replica. If one replica went down, nothing needed to change on the client\u2019s end\u2014the NDN network routed the request to a different replica. We then used this testbed setup in conjunction with GEMmaker to test NDN\u2019s usefulness. We discuss the results in the evaluation section.\nThe UI in Figure 3 provides an intuitive way to search for names that were published on the testbed. A user could create a query by selecting different components from the left-hand menu (e.g., by selecting \u201cAbsidia\u201d under genus). The user can start typing the full name, and the UI will provide a set of autocompleted names. Finally, the user could choose to view the entire name tree using the tree view. The catalog and the UI are necessary for dataset discovery; while NDN operates on names, it is more efficient and fast to discover names using a catalog. Once discovered, the names could be used for all subsequent operations. For example, once the names are known, the user can initiate retrieval or other operations, as discussed before. The API also provides a command-line interface (CLI) for name discovery, allowing the users to integrate the name discovery (and subsequent operations) with domain workflows.\nSince NDN operates on names, data naming in NDN affects application and network behaviors. The way a piece of content is named has profound impacts on content discovery, routing of user requests, data retrieval, and security. Besides, the naming of individual pieces of content seriously affects how the NDN behaves. For brevity, we do not discuss those naming trade-offs here but point the reader to our recent work on content naming in NDN.[41]\n\nIntegration with GEMmaker \nGEMmaker[19] is a workflow that takes as input a number of large RNA sequence (RNA-seq) data files and a whole genome reference file to quantify gene expression-levels underlying a specific set of experimental parameters. It produces a Gene Expression Matrix (GEM), which is a data structure that can be used to compare gene expression across the input RNA-seq samples. GEMs are commonly used in genomics and have been instrumental in several studies.[52][53][54][55] GEMmaker consists of five major steps: \n\n 1. Read RNA-seq data files as input.\n 2. Trim low-quality portions of the sequences.\n 3. Map sequences to a reference genome. \n 4. Count levels of gene expression in each RNA-seq sample.\n 5. Merge the gene expression values of each sample into a Gene Expression Matrix. \nGEMmaker is regulated by the Nextflow[56][57] workflow manager, which automates the fluid execution of each step in GEMmaker, which are defined as Nextflow processes. Nextflow has a variety of executors that enable the deployment of processes to a number of HPC and cloud environments, including Kubernetes (K8s).\nWithout NDN, the typical process a user follows to execute GEMmaker in a K8s environment is as follows. First, the user moves the whole genome reference files onto a Persistent Volume Claim (PVC), a K8s artifact that provides persistent storage to users. Nextflow is able to mount and unmount GEMmaker pods directly to this PVC, which stores all input, intermediate, and output data. When the user executes GEMmaker, Nextflow deploys pods that automatically retrieve the RNA-seq data directly from NCBI using Aspera or FTP services. One pod is submitted for each RNA sample in parallel, so ideally all RNA-seq data is downloaded simultaneously. This represents the bulk of data movement in GEMmaker. Once each sample of RNA-seq data is downloaded to the PVC, new pods are deployed to execute the next step of the workflow for each sample. Each step in GEMmaker produces a new intermediate file used as input for the next step, all of which is written to the PVC. Once GEMmaker completes, the resulting GEM can be downloaded from the PVC by the user.\nCurrently, pulling data from host-specific services is built into the GEMmaker workflow. For example, the NCBI SRA-toolkit can pull data from SRA but not all genome data repositories. With NDN, the process can be abstracted from the workflow logic as data is preloaded into the NDN testbed from any host, downloaded by name at step 1, reference data is pulled by name at step 3, and data is created and uploaded with a new name at step 5. In contrast to the typical execution of GEMmaker, with NDN, users need not retrieve the whole genome reference or the final GEM, and retrieval of RNA-seq data can occur independent of Aspera or FTP protocols, supporting a greater variety of data repositories or even locally created data (so long as a local publisher makes it available). Users need only provide the NDN names for moving data and accessing cloud APIs.\nFor GEMmaker integration, we created a NDN-capable program that pulled data from the NDN testbed using the SRA ID. We modified GEMmaker to replace the existing RNA-seq data retrieval step with this NDN retrieval program and to retrieve the whole genome references. These whole genome references are the same described previously that were generated by Pynome and cataloged in the testbed. For this small-scale testing, we added the NDN routes to the testbed machines manually. However, the NDN community provides multiple routing protocols[58] that can automate the routing updates.\n\nResults \nPerformance evaluation \nIn order to understand how caching affects data distribution, we moved datasets of different sizes between the NDN testbed and Clemson University. The SRA sequences were published into the testbed, and then the workflow modified to download data over NDN. NDN-based download only needs to know the name of the content. The rest of the infrastructure is opaque to the user.\nFor these experiments, the standard NDN tools[22] for publication and retrieval were used. Once the files were downloaded, they were utilized in the workflow. For the first experiment, we copied different sized files using existing IP based tools (wget). For comparison, we then utilized NDN-based standard tools (ndncatchunks and ndnputchunks) for the same files, with in-network caching disabled, followed by caching enabled. Each transfer experiment was repeated three times.\nThe experiments showed how NDN can improve content retrieval using in-network caching. Figure 4A shows a comparison of data download speed between NDN and HTTP. The first three bars represent three sequential downloads of a set of three SRA datasets from the NCBI repository using HTTP. The next three bars show three sequential downloads using NDN retrieval from the NDN testbed without caching. Since we manually staged the data, we knew that data source was approximately 1,500 miles away from the requester. Even then, the download performance was comparable with the HTTP performance. The real performance gain came from caching the datasets as seen in the last three bars where the first transfer was similar to HTTP and NDN without caching (two minutes) while the next two transfers only took around 20 seconds after caching kicked in. These results point toward a massive improvement opportunity since many genomics workflows download hundreds or even thousands of SRAs for a given experiment.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 4. The effect of NDN caching on DNA dataset insertion into GEMmaker workflow. A set of three NCBI SRA Arabidopsis thaliana datasets (SRR5263229:167.9 Mb, SRR5263230:167.8 Mb, SRR5263231:167.9 Mb) was sequentially transferred three times (black bars [transfer I], gray bars [transfer II], white bars [transfer III]) for insertion into the GEMmaker workflow from the NDN testbed or from the SRA repository over the internet via HTTP. The y-axis in both panels shows the transfer time of all three datasets, from request to workflow accessibility, in seconds. Error bars represent standard error of the mean. (A) The aggregate transfer times are shown for the three sequential transfers with HTTP, NDN tested without caching, and NDN with caching at 655,360 packets. (B) The effect of varying cache size (packet number in parentheses) is shown. The x-axis shows the cache capacity in packet numbers.\n\n\n\nFigure 4B shows how much in-network cache was needed to accomplish speedup using in-network caching and how the caching capacity affects the speedup. The x-axis of this figure shows the cache size in the number of NDN data packets (by default, each packet is 4,400 bytes). We find that at around 500 MB cache size, we start to see speed improvements.\nWe performed an additional experiment to better evaluate caching on data transfer in a cloud environment (Figure 5). In this caching experiment, gateway and endpoint containers were employed to determine the time it takes to download a SRA sequence dataset. The gateway container used NDN to pull the SRA sequence from a remote NDN repository and create the network cache. The endpoint container (with a cache size of 0) acted as the consumer and created a face to the gateway used to pull the data. Both the endpoint and the client were run on separate nodes to replicate real cloud scenarios. This experiment demonstrates how an NDN container can cache a dataset for use by any endpoint on the same network and indicates that an insufficiently sized content store on the gateway will prevent network caching, resulting in slower download times. This provides a basic example of how popular genomic sequences could be cached for use by multiple researchers working on the same network in the cloud.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 5. The effect of NDN caching on SRA download times on a Kubernetes cluster. A single NCBI SRA Homo sapiens kidney RNAseq dataset (SRR5139395) was downloaded a total of nine times (3 transfer trials labeled I, II, III in triplicate) for each cache size (packet number in parentheses). The following cs_max sizes (packet number) were used: 65,536 (approx 500 MB), 262,144, 327,680, 393,216, and 458,752. The Y-axis shows the download time for the dataset into the endpoint pod in seconds. Error bars represent standard error of the mean.\n\n\n\nThe actual cache sizes in the real-world would depend on request patterns as well as file sizes\u2014we are currently working on quantifying this. In any case, it is certainly feasible to utilize NDN\u2019s caching ability for very popular datasets, such as the human reference genomes. Our previous work shows that in big-science communities, even a small amount of cache significantly speeds up delivery due to the temporal locality of requests.[17]\n\nCodebase \nAll code used for these experiments are publicly available and distributed under open source licences (Table 1). NFD is the Named Data Networking forwarder that works as a software router for Interests and Data packets. ndn-cxx is an NDN library that provides the necessary API for creating NDN based \u201capps\u201d. NDN catalog is an NDN based Name lookup system\u2014an application can send an Interest to the catalog and receive the name of a dataset. The application can then utilize the name for all subsequent operations. Two versions of the catalog exists, ndn-atmos is written in C++ while ndn-python-catalog is implemented in Python.\n\r\n\n\n\n\n\n\n\n\n\n\nTable 1. Software packages used in this experiment.\n\n\n\nNextflow is a general purpose workflow manager. GEMmaker is a genomics workflow adpated for Nextflow (see Section above for more details). For these experiments, we modified GEMmaker to request data over NDN instead of standard mechanisms (such as HTTP). This modified workflow is available under this repository.\n\nDiscussion and future directions \nDiscussion \nThis work demonstrates the preliminary integration of NDN with genomics workflows. While the work shows the promise of NDN toward a simplified but capable cyberinfrastructure, several other aspects remain to be addressed before NDN can completely integrate with genomics workflows and cloud computing platforms. In this section we discuss the technical challenges as well as the economic considerations that are yet to be addressed.\n\nEconomic considerations \nNDN is a new internet architecture that operates differently that the current internet. Consequently, the users and the network operators need to consider the economic cost of moving to an NDN paradigm. This section outlines some of the economic considerations. There are two primary cost of moving to an NDN paradigm: the cost of upgrading current network equipment and the cost of storage if caching is desired.\nAs of writing this paper, NDN routers are predominately software-based. To utilize these software-based routers, the researcher needs to install NFD (an NDN packet forwarder) and NDN libraries on a machine (a server or even a laptop). All experiments in this paper were done on commodity hardware and did not require any additional capital investment. NDN is able to run as an overlay on top of the existing IP infrastructure or on Layer two circuits; in this work, we utilized NDN as an overlay on existing internet connectivity.\nAny commodity hardware (servers or desktop) with storage (depends on the workflow requirement) and a few GB memory is capable of supporting NDN. Given the current low cost of storage (an 8 TB hard drive costs around $150) even the cost of moving to a dedicated server is low. However, workflows with large storage requirement will need some capital investment. To minimize this cost and provide an alternative, we are working on two storage access mechanisms. First, when installing new storage is feasible (e.g., cost of storage continues to fall and a petabyte of storage costs around $30,000 at the time of writing this paper) it might be convenient for large research organizations to install dedicated storage that holds and serves NDN data packets; this approach improves NDN\u2019s performance since objects are already packetized and signed by the data producers. The second approach is interfacing NDN with existing storage repositories such as HTTP and FTP servers. As NDN Interests come in, they are translated into appropriate system calls (e.g., POSIX, HTTP, or FTP) and the NDN Data packets are created on-demand. This approach is slower than the first approach but does not require any additional hardware or storage, reducing the deployment cost.\nThe benefits of NDN (caching, etc.) becomes apparent when more users utilize an NDN-based network. As the networking community moves toward testbeds and deployments such as NSF-funded FABRIC[59] that incorporate NDN into their core design, the research labs and institutes connected to these networks would be able to take advantage of those infrastructures. Additionally, connecting a software NDN router to these networks and testbeds are often free (assuming an institute is already paying for internet access). These networks will create and deploy large scale in-network caches near the users and in the internet core as users continue to request data from. As users exchange data over these networks, data will be automatically cached and served from in-network caches. In the future, when ISPs deploy NDN routers, the researchers will be able to take advantage of in-network caching without added cost. However, we expect large scale ISP deployment of NDN to take a few more years.\nThe other cost is the learning and integration cost with existing workflows. This is not trivial; NDN requires careful naming considerations, aligning workflows with a name base network, and data publication. To make this process straightforward (Figure 6), we are working on containerizing several individual pieces. We hope that containerizing NDN, data transfer pods, and other components will allow researchers to simply mix-and-match different containers and integrate them with workflows without resorting to complex configuration and integration efforts. Having discussed the economic considerations, we now discuss the technical challenges that remain.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 6. Nextflow managed GEMmaker workflow on a Kubernetes Cluster.\n\n\n\nFuture directions \nSoftware performance \nWhile our work shows some attractive properties of NDN for the genomics community, there are well-known shortcomings of NDN. For example, the forwarder we used (NFD) was single-threaded and therefore its throughput is low. This has been recently addressed by a new forwarder (ndn-dpdk) that can perform forwarding at 100 Gbps. We are currently working on integrating the genomics workflow with NDN-DPDK. We hope to demonstrate further improvements as the protocol and software stack continues to mature.\n\nAccessing distributed data over NDN= \nGenomics data generation is highly distributed as data is generated in academic institutions, research labs, and the industry. While the HTDS datasets are eventually converted into in FASTQ format[60], the storage and downstream analysis data formats are highly heterogeneous with different naming systems, different storage techniques, and inconsistent and non-standard use of metadata. Seamlessly accessing these diverse datasets is an enormous challenge. Additionally, multiple types of repositories exist with different scopes: national-level (e.g., NCBI, EBI, and DDBJ) with large scale sequencing data for many organisms, community-supported repositories focused on a species or small clade of organisms, and single investigator web sites containing ad hoc datasets. Storing these datasets in different repositories that are hosted under various domain names is unlikely to scale very well for two primary reasons. First, different hosting entities utilize different schemes for data access APIs (e.g., URLs), making it necessary to understand and parse various naming schemes. Second, it is hard to find and catalog each institutional repository.\nNDN provides a scalable approach to publish and retrieve immutable content using their names in a location-agnostic fashion. NDN uses the content names for routing and request forwarding, ensuring all publicly available data can be access directly without the need for frequent housekeeping. For example, currently moving a repository under a new domain name requires a large amount of housekeeping, such as renaming the data under a new URL or linking new and old data. With NDN, the physical location of the data has no bearing on how they are named or distributed. When data is replicated, NDN brings the requests to the \u201cnearest\u201d data replica. \u201cNearest\u201d in NDN can be defined as the physically nearest replica, the most performant replica, or a combination of these (and other) factors.\nHowever, several unexplored challenges exist on applying NDN to distributed data. First, NDN operates on names. Finding these names requires the service of a third-party software or name provider. A catalog that enumerates all names under a namespace (as we discuss before) can provide this service. However, it is not yet obvious who would be responsible for running and updating the authoritative versions of these catalogs. When data is replicated, we also need to address the issue of data consistency across multiple repositories. This is still an active research direction that requires considerable attention.\n\nDistributed repositories over NDN \nA single centralized repository can become a bottleneck when subjected to a large number of queries or download requests. Moreover, it becomes a single point of failure and introduces increased latency for distant clients. NDN makes it easier to create distributed repositories since they no longer need to be tracked by their IP addresses. For example, in this work, we created a federation of three geographically distributed repositories. These repositories had to announce the prefix they intended to serve (e.g., \u201c\/genome\u201d). Even when one or more repositories go down or become unreachable, no additional maintenance is necessary; NDN routes the request to the available repositories as long as at least one repository is reachable. Similarly, when new repositories are added, the process is completely transparent to the user and does not require any action from the network administrator. However, several aspects remain to be addressed: how to allow data producers to publish long-term data efficiently, how to replicate datasets across repositories (partially or completely), and how to retrieve content most efficiently.\n\nPublication of more genomics datasets and metadata into the NDN testbed \nThere are currently hundreds of reference genomes in the NDN testbed, and we are working on updating Pynome to include more current genome builds, genomes from services other than Ensembl, and support for popular sequence aligner programs (e.g., StAR, Salmon, Kallisto, and Hisat2). We are also working on loading the metadata and SRA RNA-seq files at scale into the NDN testbed. By studying the usage logs of our integration, we will better understand the benefit NDN brings to genomics workflows. Further, the convergence of searchable metadata from multiple data repositories published in the same NDN testbed will allow for a common search and access point for genomic data.\n\nIntegration with Docker \nFor further simplification of workflows, we have created a Docker container with NDN tools, forwarder, and application libraries built-in. The resulting container is fairly lightweight. We plan to publish the image to a public repository where scientists can download and utilize the docker build \u201cas-is\u201d. These images can be deployed to a variety of cloud platforms without modifications, further simplifying the NDN access to genomics workflows.\nGEMmaker is able to run Nextflow processes inside specified containers. By adding a container that is configured with NDN to the GEMmaker directory, scripts can run inside the NDN container during a normal GEMmaker workflow. The GEMmaker workflow can then use the ndn-tools to download the SRA sequences, both from ndn-only or NDN-interfaced existing repositories. This method also provides the opportunity for decreased data retrieval time due to NDN in-network caching and allows GEMMaker to benefit from all the NDN features we described earlier.\n\nIntegration With Kubernetes \nThe genomics community is moving toward a cloud-based model. Container orchestration platforms (such as Kubernetes) are more commonly being used in favor of traditional HPC clusters. We believe that enabling users to easily move data from an NDN network to a Kubernetes cluster is imperative for the widespread adoption of this use case.\nTo achieve this goal, we are engineering the integration of NDN with a Data Transfer Pod (DTP) that runs as a deployment of different containers that enable users to read and write data to a Kubernetes cluster. A DTP is a configurable collection of containers, each representing a different data transfer protocol\/interface. A DTP uses the officially maintained images of each protocol\/interface, removing the need for integration, aside from adding the container to the DTP deployment, which is an almost identical process for each protocol\/interface. This makes adding new protocols very simple, as there is no need to build a custom image for each protocol\/interface, as long as an image already exists. Almost all interfaces\/protocols used by the community have officially maintained images associated with them.\nThe DTP tool will allow Kubernetes users to easily access data stored on NDN-based repositories. Each DTP container provides the client with a mechanism to utilize a different data transfer method (e.g., NDN, Aspera, Globus, S3). Using a DTP aims to address the need for simple access to data from a variety of sources. A DTP is not coupled with any particular workflow, so users will be able to pull or push NDN data as a pre- or post-processing step of their workflow, without modifying the workflow itself. The DTP can also be used to pull data from other sources if it is not present in an NDN framework.\nWe are modifying the existing GEMmaker workflow to use an NDN via the DTP that queries the NDN framework for the required reference genome files and SRA datasets. If the SRA dataset file exists in the NDN framework, the DTP will pull the data onto a Kubernetes persistent volume claim (PVC). For example, we are adapting NDN to work with the Pacific Research Platform (PRP)[61] Nautilus Kubernetes cluster (6). If the user knows the content name (or metadata) but content does not exist in NDN format, the DTP will pull the dataset from SRA with Aspera and publish it in the NDN testbed and then pull into PRP. Once published, the dataset will now exist in the NDN testbed and benefit from NDN attributes, including caching. Once the DTP completes its job, the GEMmaker workflow will function in the same way it does now so no new code needs to be written. We are also developing a cache retention policy to allow the SRA files to evaporate if they are not accessed after a certain period of time.\n\nLimitations \nThe NDN prototype we used (NFD) and other components we used (catalogs and repo) are research prototypes. The performance and scalability of these prototypes are being improved by the NDN community. Additionally, utilization of NDN containers on PRP has not been explored before; we are working on optimizing both the containers and their interactions with the cloud platforms. We are also working on better understanding the caching and storage needs of the genomics community by looking at real-world request patterns and object sizes associated with them.\n\nConclusion \nIn this paper, we enumerate the cyberinfrastruture challenges faced by the genomics community. We discuss NDN, a novel but well-researched future internet architecture that can address these challenges at the network layer. We present our efforts in integrating NDN with a genomics workflow, GEMmaker. We describe the creation of an NDN-complaint naming scheme that is also acceptable to the genomics community. We find that genomics names are already hierarchical and easily translated into NDN names. We publish the actual datasets into an NDN testbed and show that NDN can serve data from anywhere, simplifying data management. Finally, through the integration with GEMmaker, we show NDN\u2019s in-network caching can speed up data retrieval and insertion into the workflow by six times.\n\nAcknowledgements \nAuthor contributions \nFF, SS, and SF are the PIs who designed the study, prepared and edited the manuscript, and provided guidance to the students and programmers. CO, DR, CM, TB, and RP are the students and programmers who performed GEMmaker integration, deployed the code and prototype on Pacific Research Platform and evaluated it. They also helped writing the manuscript.\n\nData availability statement \nThe datasets presented in this study can be found in online repositories. The names of the repository\/repositories and accession number(s) can be found in the article.\n\nFunding \nThis work was supported by National Science Foundation Award #1659300 \u201cCC*Data: National Cyberinfrastructure for Scientific Data Analysis at Scale (SciDAS),\u201d National Science Foundation Award #2019012 \u201cCC* Integration-Large: N-DISE: NDN for Data Intensive Science Experiments,\u201d and Tennessee Tech Faculty Research Grant. National Science Foundation Award #2019163 CC* Integration-Small: Error Free File Transfer for Big Science.\n\nConflict of interest \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 Cinquini, L.; Chrichton, D.; Mattmann, C. et al. (2014). \"The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data\". Future Generation Computer Systems 36: 400\u201317. doi:10.1016\/j.future.2013.07.002.   \n\n\u2191 ATLAS Collaboration; Aad, G.; Abat, E. et al. (2008). \"The ATLAS Experiment at the CERN Large Hadron Collider\". Journal of Instrumentation 3: S08003. doi:10.1088\/1748-0221\/3\/08\/S08003.   \n\n\u2191 Dewdney, P.E.; Hall, P.J.; Schillizzi, R.T. et al. (2009). \"The Square Kilometre Array\". Proceedings of the IEEE 97 (8): 1482-1496. doi:10.1109\/JPROC.2009.2021005.   \n\n\u2191 LSST Dark Energy Science Collaboration; Abate, A.; Aldering, G. et al. (2012). \"Large Synoptic Survey Telescope Dark Energy Science Collaboration\". arXiv: 1\u2013133. https:\/\/arxiv.org\/abs\/1211.0310v1 .   \n\n\u2191 5.0 5.1 Sayers, E.W.; Beck, J.; Brister, J.R. et al. (2020). \"Database resources of the National Center for Biotechnology Information\". Nucleic Acids Research (D1): D9\u2013D16. doi:10.1093\/nar\/gkz899. PMC PMC6943063. PMID 31602479. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6943063 .   \n\n\u2191 Tsuchiya, S.; Sakamoto, Y.; Tsuchimoto, Y. et al. (2012). \"Big Data Processing in Cloud Environments\" (PDF). Fujitsu Scientific & Technical Journal 48 (2): 159\u201368. https:\/\/www.fujitsu.com\/downloads\/MAG\/vol48-2\/paper09.pdf .   \n\n\u2191 Luo, J.; Wu, M.; Gopukumar, D. et al. (2016). \"Big Data Application in Biomedical Research and Health Care: A Literature Review\". Biomedical Informatics Insights 8: 1\u201310. doi:10.4137\/BII.S31559. PMC PMC4720168. PMID 26843812. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720168 .   \n\n\u2191 8.0 8.1 8.2 Shannigrahi, S.; Fan, C.; Papadopoulos, C. et al. (2018). \"NDN-SCI for managing large scale genomics data\". Proceedings of the 5th ACM Conference on Information-Centric Networking: 204\u201305. doi:10.1145\/3267955.3269022.   \n\n\u2191 McCombie, W.R.; McPherson, J.D.; Mardis, E.R. (2016). \"Next-Generation Sequencing Technologies\". Cold Spring Harbor Perspectives in Medicine 9 (11): 1\u201310. doi:10.4137\/BII.S31559. PMC PMC4720168. PMID 26843812. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720168 .   \n\n\u2191 National Human Genome Research Institute (2020). \"The Cost of Sequencing a Human Genome\". National Institutes of Health. https:\/\/www.genome.gov\/about-genomics\/fact-sheets\/Sequencing-Human-Genome-cost . Retrieved 16 March 2020 .   \n\n\u2191 Lowy-Gallego, E.; Fairley, S.; Zheng-Bradley, X. et al. (2019). \"Variant calling on the GRCh38 assembly with the data from phase three of the 1000 Genomes Project\". Wellcome Open Research 4: 50. doi:10.12688\/wellcomeopenres.15126.2. PMC PMC7059836. PMID 32175479. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7059836 .   \n\n\u2191 12.0 12.1 NCBI (2020). \"Sequence Read Archive\". NCBI. https:\/\/trace.ncbi.nlm.nih.gov\/Traces\/sra\/ . Retrieved 04 February 2020 .   \n\n\u2191 13.0 13.1 Stephens, Z.D.; Lee, S.Y.; Faghri, F. et al. (2015). \"Big Data: Astronomical or Genomical?\". PLoS Biology 13 (7): e1002195. doi:10.1371\/journal.pbio.1002195. PMC PMC4494865. PMID 26151137. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4494865 .   \n\n\u2191 Olschanowsky, C.; Shannigrahi, S. Papadopolous, C. et al. (2014). \"Supporting climate research using named data networking\". Proceedings of the IEEE 20th International Workshop on Local & Metropolitan Area Networks: 1\u20136. doi:10.1109\/LANMAN.2014.7028640.   \n\n\u2191 15.0 15.1 15.2 15.3 Fan, C.; Shannigrahi, S.; DiBenedetto, S. et al. (2015). \"Managing scientific data with named data networking\". Proceedings of the Fifth International Workshop on Network-Aware Data Management: 1\u20137. doi:10.1145\/2832099.2832100.   \n\n\u2191 16.0 16.1 Shannigrahi, S.; Papadopolous, C.; Yeh, E. et al. (2015). \"Named Data Networking in Climate Research and HEP Applications\". Journal of Physics: Conference Series 664: 052033. doi:10.1088\/1742-6596\/664\/5\/052033.   \n\n\u2191 17.0 17.1 17.2 Shannigrahi, S.; Fan, C.; Papadopolous, C. (2017). \"Request aggregation, caching, and forwarding strategies for improving large climate data distribution with NDN: a case study\". Proceedings of the 4th ACM Conference on Information-Centric Networking: 54\u201365. doi:10.1145\/3125719.3125722.   \n\n\u2191 Shannigrahi, S.; Fan, C.; Papadopolous, C. et al. (2018). \"Named Data Networking Strategies for Improving Large Scientific Data Transfers\". Proceedings of the 2018 IEEE International Conference on Communications Workshops. doi:10.1109\/ICCW.2018.8403576.   \n\n\u2191 19.0 19.1 Hadish, J.; Biggs, T.; Shealy, B. et al. (22 January 2020). \"SystemsGenetics\/GEMmaker: Release v1.1\". Zenodo. doi:10.5281\/zenodo.3620945. https:\/\/zenodo.org\/record\/3620945 .   \n\n\u2191 20.0 20.1 Zhang, L.; Afanasyev, A.; Burke, J. et al. (2014). \"Named data networking\". ACM SIGCOMM Computer Communication Review 44 (3): 66-73. doi:10.1145\/2656877.2656887.   \n\n\u2191 Hoque, A.K.M.M.; Amin, S.O.; Alyyan, A. et al. (2013). \"NLSR: Named-data link state routing protocol\". Proceedings of the 3rd ACM SIGCOMM workshop on Information-centric networking: 15\u201320. doi:10.1145\/2491224.2491231.   \n\n\u2191 22.0 22.1 Afanasyev, A.; Shi, J.; Zhang, B. et al. (October 2016). \"NFD Developer\u2019s Guide - Revision 7\". NDN, Technical Report NDN-0021. https:\/\/named-data.net\/publications\/techreports\/ndn-0021-7-nfd-developer-guide\/ .   \n\n\u2191 So, W.; Narayanan, A.; Oran, D. et al. (2013). \"Named data networking on a router: Forwarding at 20gbps and beyond\". ACM SIGCOMM Computer Communication Review 43 (4): 495\u20136. doi:10.1145\/2534169.2491699.   \n\n\u2191 Khoussi, S.; Nouri, A.; Shi, J. et al. (2019). \"Performance Evaluation of the NDN Data Plane Using Statistical Model Checking\". Proceedings of the International Symposium on Automated Technology for Verification and Analysis: 534\u201350. doi:10.1007\/978-3-030-31784-3_31.   \n\n\u2191 Chen, S.; Shi, W; Cao, A. et al. (2014). \"NDN Repo: An NDN Persistent Storage Model\" (PDF). Tsinghua University. http:\/\/learn.tsinghua.edu.cn:8080\/2011011088\/WeiqiShi\/content\/NDNRepo.pdf .   \n\n\u2191 \"DDBJ\". Research Organization of Information and Systems. 2019. https:\/\/www.ddbj.nig.ac.jp\/index-e.html . Retrieved 21 October 2019 .   \n\n\u2191 \"EMBL-EBI\". Wellcome Genome Campus. 2021. https:\/\/www.ebi.ac.uk\/ . Retrieved 02 February 2021 .   \n\n\u2191 Lathe, W.; williams, J.; Mangan, M. et al. (2008). \"Genomic Data Resources: Challenges and Promises\". Nature Education 1 (3): 2. https:\/\/www.nature.com\/scitable\/topicpage\/genomic-data-resources-challenges-and-promises-743721\/ .   \n\n\u2191 Dankar, F.K.; Ptisyn, A.; Dankar, S.K. (2018). \"The development of large-scale de-identified biomedical databases in the age of genomics\u2014principles and challenges\". Human Genomics 12: 19. doi:10.1186\/s40246-018-0147-5. PMC PMC5894154. PMID 29636096. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5894154 .   \n\n\u2191 National Genomics Data Center Members and Partners (2020). \"Database Resources of the National Genomics Data Center in 2020\". Nucleic Acids Research 48 (D1): D24\u201333. doi:10.1093\/nar\/gkz913. PMC PMC7145560. PMID 31702008. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7145560 .   \n\n\u2191 The LHC Study Group (May 1991). \"Design Study of the Large Hadron Collider (LHC): A Multiparticle Collider in the LEP Tunnel\". CERN. https:\/\/cds.cern.ch\/record\/220493 .   \n\n\u2191 Taylor, K.E.; Doutriaux, C. (28 April 2010). \"CMIP5 Model Output Requirements: File Contents and Format, Data Structure and Metadata\" (PDF). PCMDI. https:\/\/pcmdi.llnl.gov\/mips\/cmip5\/docs\/CMIP5_output_metadata_requirements28Apr10.pdf?id=89 .   \n\n\u2191 Munson, M.C.; Simu, S. (11 November 2013). \"Bulk data transfer\". ScienceON. https:\/\/scienceon.kisti.re.kr\/srch\/selectPORSrchPatent.do?cn=USP2015038996945 .   \n\n\u2191 \"Aspera\". IBM. https:\/\/www.ibm.com\/products\/aspera .   \n\n\u2191 \"Globus\". University of Chicago. https:\/\/www.globus.org\/ . Retrieved 02 February 2021 .   \n\n\u2191 Rajasekar, A.; Moore, R.; Hou, C.-Y. et al. (2010). \"iRODS Primer: Integrated Rule-Oriented Data System\". Synthesis Lectures on Information Concepts, Retrieval, and Services 2: 1\u2013143. doi:10.2200\/S00233ED1V01Y200912ICR012.   \n\n\u2191 Chiang, G.-T.; Clapham, P.; Qi, G. et al. (2011). \"Implementing a genomic data management system using iRODS in the Wellcome Trust Sanger Institute\". BMC Bioinformatics 12: 361. doi:10.1186\/1471-2105-12-361. PMC PMC3228552. PMID 21906284. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3228552 .   \n\n\u2191 Eilbeck, K.; Lewis, S.E.; Mungall, C.J. et al. (2005). \"The Sequence Ontology: A tool for the unification of genome annotations\". Genome Biology 6 (5): R44. doi:10.1186\/gb-2005-6-5-r44. PMC PMC1175956. PMID 15892872. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1175956 .   \n\n\u2191 Schriml, L.M.; Arze, C.; Nadendla, S. et al. (2012). \"Disease Ontology: A backbone for disease semantic integration\". Nucleic Acids Research 40 (DB 1): D940-6. doi:10.1093\/nar\/gkr972. PMC PMC3245088. PMID 22080554. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3245088 .   \n\n\u2191 Gene Ontology Consortium (2015). \"Gene Ontology Consortium: Going forward\". Nucleic Acids Research 43 (DB 1): D1049\u201356. doi:10.1093\/nar\/gku1179. PMC PMC4383973. PMID 25428369. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383973 .   \n\n\u2191 41.0 41.1 Shannigrahi, S.; Fan, C.; Partridge, C. (2020). \"What's in a Name?: Naming Big Science Data in Named Data Networking\". Proceedings of the 7th ACM Conference on Information-Centric Networking: 12\u201323. doi:10.1145\/3405656.3418717.   \n\n\u2191 Shannigrahi, S. (Summer 2019). \"The Future of Networking Is the Future of Big Data\". Colorado State University. https:\/\/mountainscholar.org\/bitstream\/handle\/10217\/197325\/Shannigrahi_colostate_0053A_15549.pdf?sequence=1&isAllowed=y .   \n\n\u2191 43.0 43.1 43.2 Tierney, B.; Metzger, J. (July 2010). \"Higher Performance Bulk Data Transfer\". ESnet. https:\/\/fasterdata.es.net\/assets\/fasterdata\/JT-201010.pdf . Retrieved 02 February 2021 .   \n\n\u2191 Mills, N.; Bensman, E.M.; Poehlman, W.L. et al. (2019). \"Moving Just Enough Deep Sequencing Data to Get the Job Done\". Bioinformatics and Biology Insights 13: 1177932219856359. doi:10.1177\/1177932219856359. PMC PMC6572328. PMID 31236009. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6572328 .   \n\n\u2191 Yu, Y.; Afanasyev, A.; Clark, D. et al. (2015). \"Schematizing Trust in Named Data Networking\". Proceedings of the 2nd ACM Conference on Information-Centric Networking: 177\u2013186. doi:10.1145\/2810156.2810170.   \n\n\u2191 \"ESnet\". ESnet. https:\/\/fasterdata.es.net\/ . Retrieved 02 February 2021 .   \n\n\u2191 \"Welcome to ICANN!\". ICANN. https:\/\/www.icann.org\/resources\/pages\/welcome-2012-02-25-en . Retrieved 14 November 2020 .   \n\n\u2191 \"CC*Data: National Cyberinfrastructure for Scientific Data Analysis at Scale (SciDAS)\". National Science Foundation. 24 October 2019. https:\/\/www.nsf.gov\/awardsearch\/showAward?AWD_ID=1659300 . Retrieved 02 February 2021 .   \n\n\u2191 Zerbino, D.R.; Achuthan, P.; Akanni, W. et al. (2018). \"Ensembl 2018\". Nucleic Acids Research 46 (D1): D754\u201361. doi:10.1093\/nar\/gkx1098. PMC PMC5753206. PMID 29155950. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5753206 .   \n\n\u2191 \"SystemsGenetics\/pynome\". GitHub. https:\/\/github.com\/SystemsGenetics\/pynome . Retrieved 02 February 2021 .   \n\n\u2191 Olschanowsky, C.; Shannigrahi, S.; Papadopoulos, C. (2014). \"Supporting climate research using named data networking\". Proceedings of the 2014 IEEE 20th International Workshop on Local & Metropolitan Area Networks: 1\u20136. doi:10.1109\/LANMAN.2014.7028640.   \n\n\u2191 Ficklin, S.P.; Dunwoodie, L.J.; Poehlman, W.L. et al. (2017). \"Discovering Condition-Specific Gene Co-Expression Patterns Using Gaussian Mixture Models: A Cancer Case Study\". Scientific Reports 7 (1): 8617. doi:10.1038\/s41598-017-09094-4. PMC PMC5561081. PMID 28819158. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5561081 .   \n\n\u2191 Roche, K.; Feltus, F.A.; Park, J.P. et al. (2017). \"Cancer cell redirection biomarker discovery using a mutual information approach\". PLoS One 12 (6): e0179265. doi:10.1371\/journal.pone.0179265. PMC PMC5464651. PMID 28594912. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5464651 .   \n\n\u2191 Dunwoodie, L.J.; Poehlman, W.L.; Ficklin, S.P. et al. (2018). \"Discovery and validation of a glioblastoma co-expressed gene module\". Oncotarget 9 (13): 10995-11008. doi:10.18632\/oncotarget.24228. PMC PMC5834250. PMID 29541392. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5834250 .   \n\n\u2191 Poehlman, W.L.; Hsieh, J.J.; Feltus, F.A. et al. (2019). \"Linking Binary Gene Relationships to Drivers of Renal Cell Carcinoma Reveals Convergent Function in Alternate Tumor Progression Paths\". Scientific Reports 9 (1): 2899. doi:10.1038\/s41598-019-39875-y. PMC PMC6393532. PMID 30814637. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6393532 .   \n\n\u2191 \"Nextflow\". Seqera Labs. https:\/\/nextflow.io\/ . Retrieved 21 October 2019 .   \n\n\u2191 Di Tommaso, P.D.; Chatzou, M.; Floden, E.W. et al. (2017). \"Nextflow enables reproducible computational workflows\". Nature Biotechnology 35 (4): 316\u201319. doi:10.1038\/nbt.3820. PMID 28398311.   \n\n\u2191 Wang, Y.; Li, Z.; Tyson, G. et al. (2013). \"Optimal cache allocation for Content-Centric Networking\". Proceedings of the 21st IEEE International Conference on Network Protocols: 1\u201310. doi:10.1109\/ICNP.2013.6733577.   \n\n\u2191 \"FABRIC\". RENCI. https:\/\/fabric-testbed.net\/ . Retrieved 14 November 2020 .   \n\n\u2191 Cock, P.J.A.; Fields, C.J.; Goto, N. et al. (2010). \"The Sanger FASTQ file format for sequences with quality scores, and the Solexa\/Illumina FASTQ variants\". Nucleic Acids Research 38 (6): 1767-71. doi:10.1093\/nar\/gkp1137. PMC PMC2847217. PMID 20015970. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2847217 .   \n\n\u2191 Smarr, L.; Crittenden, C.; DeFanti, T. et al. (2018). \"The Pacific Research Platform: Making High-Speed Networking a Reality for the Scientist\". Proceedings of the Practice and Experience on Advanced Research Computing: 1\u20138. doi:10.1145\/3219104.3219108.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original paper listed references alphabetically; this wiki lists them by order of appearance, by design. The two footnotes were turned into inline references for convenience.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\">https:\/\/www.limswiki.org\/index.php\/Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on big dataLIMSwiki journal articles on cloud computingLIMSwiki journal articles on genome informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 22 March 2021, at 19:38.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 453 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","784b63abe742a2cc6286e50d625220ec_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Named_data_networking_for_genomics_data_management_and_integrated_workflows skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Named data networking for genomics data management and integrated workflows<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Advanced <a href=\"https:\/\/www.limswiki.org\/index.php\/Imaging_informatics\" title=\"Imaging informatics\" class=\"wiki-link\" data-key=\"fc0ae6a154d8896767defefdb6d14d0e\">imaging<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/DNA_sequencing\" title=\"DNA sequencing\" class=\"wiki-link\" data-key=\"7ff86b38049c37e30858efd13bd00925\">DNA sequencing<\/a> technologies now enable the diverse biology community to routinely generate and analyze terabytes of high-resolution biological data. The community is rapidly heading toward the petascale in single-investigator <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> settings. As evidence, the National Center for Biotechnology Information (NCBI) Sequence Read Archive (SRA) central DNA sequence repository alone contains over 45 petabytes of biological data. Given the geometric growth of this and other <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomics<\/a> repositories, an exabyte of mineable biological data is imminent. The challenges of effectively utilizing these datasets are enormous, as they are not only large in size but also stored in various geographically distributed repositories such as those hosted by the NCBI, as well as in the DNA Data Bank of Japan (DDBJ), European Bioinformatics Institute (EBI), and NASA\u2019s GeneLab. \n<\/p><p>In this work, we first systematically point out the <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a> challenges of the genomics community. We then introduce named data networking (NDN), a novel but well-researched internet architecture capable of solving these challenges at the network layer. NDN performs all operations such as forwarding requests to data sources, content discovery, access, and retrieval using content names (that are similar to traditional filenames or filepaths), all while eliminating the need for a location layer (the IP address) for data management. Utilizing NDN for genomics workflows simplifies data discovery, speeds up data retrieval using in-network caching of popular datasets, and allows the community to create infrastructure that supports operations such as creating federation of content repositories, retrieval from multiple sources, remote data subsetting, and others. Using name-based operations also streamlines deployment and integration of workflows with various <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud<\/a> platforms. \n<\/p><p>We make four signigicant contributions with this work. First, we enumerate the cyberinfrastructure challenges of the genomics community that NDN can alleviate. Second, we describe our efforts in applying NDN for a contemporary genomics <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a> (GEMmaker) and quantify the improvements. The preliminary evaluation shows a sixfold speed up in data insertion into the workflow. Third, as a pilot, we have used an NDN naming scheme (agreed upon by the community and discussed in the \"Method\" section) to publish data from broadly used data repositories, including the NCBI SRA. We have loaded the NDN testbed with these pre-processed genomes that can be accessed over NDN and used by anyone interested in those datasets. Finally, we discuss our continued effort in integrating NDN with cloud computing platforms, such as the Pacific Research Platform (PRP). \n<\/p><p>The reader should note that the goal of this paper is to introduce NDN to the genomics community and discuss NDN\u2019s properties that can benefit the genomics community. We do not present an extensive performance evaluation of NDN; we are working on extending and evaluating our pilot deployment and will present systematic results in a future work.\n<\/p><p><b>Keywords<\/b>: genomics data, genomics workflows, large science data, cloud computing, named data networking\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Scientific communities are entering a new era of exploration and discovery in many fields, driven by high-density data accumulation. A few examples are climate science<sup id=\"rdp-ebb-cite_ref-CinquiniTheEarth14_1-0\" class=\"reference\"><a href=\"#cite_note-CinquiniTheEarth14-1\">[1]<\/a><\/sup>, high-energy particle physics (HEP)<sup id=\"rdp-ebb-cite_ref-AtlasCollabTheAtlas08_2-0\" class=\"reference\"><a href=\"#cite_note-AtlasCollabTheAtlas08-2\">[2]<\/a><\/sup>, astrophysics<sup id=\"rdp-ebb-cite_ref-DewdneyTheSquare09_3-0\" class=\"reference\"><a href=\"#cite_note-DewdneyTheSquare09-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AbateLarge12_4-0\" class=\"reference\"><a href=\"#cite_note-AbateLarge12-4\">[4]<\/a><\/sup>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomics<\/a><sup id=\"rdp-ebb-cite_ref-SayersData20_5-0\" class=\"reference\"><a href=\"#cite_note-SayersData20-5\">[5]<\/a><\/sup>, seismology<sup id=\"rdp-ebb-cite_ref-TsuchiyaBigData12_6-0\" class=\"reference\"><a href=\"#cite_note-TsuchiyaBigData12-6\">[6]<\/a><\/sup>, and biomedical research<sup id=\"rdp-ebb-cite_ref-LuoBigData16_7-0\" class=\"reference\"><a href=\"#cite_note-LuoBigData16-7\">[7]<\/a><\/sup>, just to name a few. Often referred to as \u201cdata-intensive\u201d science, these communities utilize and generate extremely large volumes of data, often reaching into the petabytes<sup id=\"rdp-ebb-cite_ref-ShannigrahiNDN18_8-0\" class=\"reference\"><a href=\"#cite_note-ShannigrahiNDN18-8\">[8]<\/a><\/sup> and soon projected to reach into the exabytes.\n<\/p><p>Data-intensive science has created radically new opportunities. Take for example high-throughput <a href=\"https:\/\/www.limswiki.org\/index.php\/DNA_sequencing\" title=\"DNA sequencing\" class=\"wiki-link\" data-key=\"7ff86b38049c37e30858efd13bd00925\">DNA sequencing<\/a> (HTDS). Until recently, HTDS was slow and expensive, and only a few institutes were capable of performing it at scale.<sup id=\"rdp-ebb-cite_ref-MccombieNext19_9-0\" class=\"reference\"><a href=\"#cite_note-MccombieNext19-9\">[9]<\/a><\/sup> With the advances in supercomputers, specialized DNA sequencers, and better <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> algorithms, the effectiveness and cost of sequencing has dropped considerably and continues to drop. For example, sequencing the first reference human genome cost around $2.7 billion over 15 years, while currently it costs under $1,000 to resequence a human genome.<sup id=\"rdp-ebb-cite_ref-NIHTheCost20_10-0\" class=\"reference\"><a href=\"#cite_note-NIHTheCost20-10\">[10]<\/a><\/sup> With commercial incentives, several companies are offering fragmented genome re-sequencing under $100, performed in only a few days. This massive drop in cost and improvement in speed supports more advanced scientific discovery. For example, earlier scientists could only test their hypothesis on a small number of genomes or gene expression conditions within or between species. With more publicly available datasets<sup id=\"rdp-ebb-cite_ref-SayersData20_5-1\" class=\"reference\"><a href=\"#cite_note-SayersData20-5\">[5]<\/a><\/sup>, scientists can test their hypotheses against a larger number of genomes, potentially enabling them to identify rare mutations, precisely classify diseases based on a specific patient, and, thusly, more accurately treat the disease.<sup id=\"rdp-ebb-cite_ref-Lowy-GallegoVariant19_11-0\" class=\"reference\"><a href=\"#cite_note-Lowy-GallegoVariant19-11\">[11]<\/a><\/sup>\n<\/p><p>While the growth of DNA sequencing is encouraging, it has also created difficulty in genomics <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a>. For example, the National Center for Biotechnology Information\u2019s (NCBI) Sequence Read Archive (SRA) database hosts 42 petabytes of publicly accessible DNA sequence data.<sup id=\"rdp-ebb-cite_ref-NCBISequence20_12-0\" class=\"reference\"><a href=\"#cite_note-NCBISequence20-12\">[12]<\/a><\/sup> Scientists desiring to use public data must discover (or locate) the data and move it from globally distributed sites to on-premize clusters and distributed computing platforms, including public and commercial <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">clouds<\/a>. Public repositories such as the NCBI SRA contain a subset of all available genomics data.<sup id=\"rdp-ebb-cite_ref-StephensBigData15_13-0\" class=\"reference\"><a href=\"#cite_note-StephensBigData15-13\">[13]<\/a><\/sup> Similar repositories are hosted by NASA, the <a href=\"https:\/\/www.limswiki.org\/index.php\/National_Institutes_of_Health\" title=\"National Institutes of Health\" class=\"wiki-link\" data-key=\"e5c215c48e73ae58b0695dc2af951cd0\">National Institutes of Health<\/a> (NIH), and other organizations. Even though these datasets are highly curated, each public repository uses their own standards for data naming, retrieval, and discovery that makes locating and utilizing these datasets difficult.\n<\/p><p>Moreover, data management problems require the community to build and the scientists to spend time learning complex infrastructures (e.g., cloud platforms, grids) and creating tools, scripts, and workflows that can (semi-) automate their research. The current trend of moving from localized institutional storage and computing to an on-demand cloud computing model adds another layer of complexity to the workflows. The next generation of scientific breakthroughs may require massive data. Our ability to manage, distribute, and utilize these types of extreme-scale datasets and securely integrate them with computational platforms may dictate our success (or failure) in future scientific research.\n<\/p><p>Our experience in designing and deploying protocols for \"big data\" science<sup id=\"rdp-ebb-cite_ref-ShannigrahiNDN18_8-1\" class=\"reference\"><a href=\"#cite_note-ShannigrahiNDN18-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OlschanowskySupport14_14-0\" class=\"reference\"><a href=\"#cite_note-OlschanowskySupport14-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FanManaging15_15-0\" class=\"reference\"><a href=\"#cite_note-FanManaging15-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShannigrahiNamed15_16-0\" class=\"reference\"><a href=\"#cite_note-ShannigrahiNamed15-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShannigrahiRequest17_17-0\" class=\"reference\"><a href=\"#cite_note-ShannigrahiRequest17-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShannigrahiNamed18_18-0\" class=\"reference\"><a href=\"#cite_note-ShannigrahiNamed18-18\">[18]<\/a><\/sup> suggests that using hierarchical and community-developed names for storing, discovering, and accessing data can dramatically simplify <a href=\"https:\/\/www.limswiki.org\/index.php\/Scientific_data_management_system\" title=\"Scientific data management system\" class=\"wiki-link\" data-key=\"9f38d322b743f578fef487b6f3d7c253\">scientific data management systems<\/a> (SDMSs), and that the network is the ideal place for integrating domain <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a> with distributed services. In this work, we propose a named ecosystem over an evolving but well-researched future internet architecture: named data networking (NDN). NDN utilizes content names for all data management operations such as content addressing, content discovery, and retrieval. Utilizing content names for all network operations massively simplifies data management infrastructure. Users simply ask for the content by name (e.g., \u201c\/ncbi\/homo\/sapiens\/hg38\u201d) and the network delivers the content to the user.\n<\/p><p>Using content names that are understood by the end-user over an NDN network provides multiple advantages: natural caching of popular content near the users, unified access mechanisms, and location-agnostic publication of data and services. For example, a dataset properly named can be downloaded by, for example, NCBI or GeneLab at NASA, whichever is closer to the researcher. Additionally, the derived data (results, annotations, publications) are easily publishable into the network (possibly after vetting and quality control by NCBI or NASA) and immediately discoverable if appropriate naming conventions are agreed upon and followed. Finally, NDN shifts the trust to content itself; each piece of content is <a href=\"https:\/\/www.limswiki.org\/index.php\/Encryption\" title=\"Encryption\" class=\"wiki-link\" data-key=\"86a503652ed5cc9d8e2b0252a480b5e1\">cryptographically<\/a> signed by the data producer and verifiable by anyone for provenance.\n<\/p><p>In this work, we first introduce NDN and the architectural constructs that make it attractive for the genomics community. We then discuss the data management and cyberinfrastructure challenges faced by the genomics community and how NDN can help alleviate them. We then present our pilot study applying NDN to a contemporary genomics workflow GEMmaker<sup id=\"rdp-ebb-cite_ref-HadishSystems20_19-0\" class=\"reference\"><a href=\"#cite_note-HadishSystems20-19\">[19]<\/a><\/sup> and evaluate the integration. Finally, we discuss future research directions and an integration roadmap with cloud computing services.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Named_data_networking\">Named data networking<\/span><\/h2>\n<p>NDN<sup id=\"rdp-ebb-cite_ref-ZhangNamed14_20-0\" class=\"reference\"><a href=\"#cite_note-ZhangNamed14-20\">[20]<\/a><\/sup> is a new networking paradigm that adopts a drastically different communication model than that current IP model. In NDN, data is accessed by content names (e.g., \u201c\/Human\/DNA\/Genome\/hg38\u201d) rather than through the host where it resides (e.g., <a rel=\"external_link\" class=\"external free\" href=\"ftp:\/\/ftp.ncbi.nlm.nih.gov\/refseq\/H_sapiens\/annotation\/GRCh38_latest\/refseq_identifiers\/GRCh38_latest_genomic.fna.gz\" target=\"_blank\">ftp:\/\/ftp.ncbi.nlm.nih.gov\/refseq\/H_sapiens\/annotation\/GRCh38_latest\/refseq_identifiers\/GRCh38_latest_genomic.fna.gz<\/a>). Naming the data allows the network to participate in operations that were not feasible before. Specifically, the network can take part in discovering and local caching of the data, merging similar requests, retrieval from multiple distributed data sources, and more. In NDN, the communication primitive is straightforward (Figure 1): the consumer asks for the content by content name (an \u201cInterest\u201d in NDN terminology), and the network forwards the request toward the publisher.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Ogle_FrontBigData2021_4.jpg\" class=\"image wiki-link\" data-key=\"38f3cdc837968a52a1db8b211b393f33\"><img alt=\"Fig1 Ogle FrontBigData2021 4.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2e\/Fig1_Ogle_FrontBigData2021_4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 1.<\/b> NDN Forwarding. The two servers on the right announce a namespace (\/google) for the data they serve. The routers make a note of this incoming announcement. When the laptops ask for \/google\/index.html, the routers forward the requests on the appropriate interfaces (31, 32, or both, depending on configuration). Data follows the reverse path.<sup id=\"rdp-ebb-cite_ref-ZhangNamed14_20-1\" class=\"reference\"><a href=\"#cite_note-ZhangNamed14-20\">[20]<\/a><\/sup><\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For communication, NDN uses two types of packets: Interest and Data. The content consumer initiates communication in NDN. To retrieve data, a consumer sends out an Interest packet into the network, which carries a name that identifies the desired data. One such content name (similar to a resource identifier) might be \u201c\/google\/index.html\u201d. A network router maintains a name-based forwarding table (FIB) (Figure 1). The router remembers the interface from which the request arrives and then forwards the Interest packet by looking up the name in its FIB. FIBs are populated using a name-based routing protocol such as Named-data Link State Routing Protocol (NLSR).<sup id=\"rdp-ebb-cite_ref-Hoque_NLSR13_21-0\" class=\"reference\"><a href=\"#cite_note-Hoque_NLSR13-21\">[21]<\/a><\/sup>\n<\/p><p>NDN routes and forwards packets based on content names<sup id=\"rdp-ebb-cite_ref-AfanasyevNFD16_22-0\" class=\"reference\"><a href=\"#cite_note-AfanasyevNFD16-22\">[22]<\/a><\/sup>, which eliminates various problems that addresses pose in the IP architecture, such as address space exhaustion, Network Address Translation (NAT) traversal, mobility, and address management. In NDN, routers perform component-based longest prefix match of the Interest name the FIB. Routing in NDN is similar to IP routing. Instead of announcing IP prefixes, an NDN router announces name prefixes that it is willing to serve (e.g., \u201c\/google\u201d). The announcement is propagated through the network and eventually populates the FIB of every router. Routers match incoming Interests against the FIB using longest prefix match. For example, \u201c\/google\/videos\/movie1.mpg\u201d might match \u201c\/google\u201d or \u201c\/google\/video\u201d. Though an unbounded namespace raises the question of how to maintain control over the routing table sizes and whether looking up variable-length, hierarchical names can be done at line rate, previous works have shown that it is indeed possible to forward packets at 100 Gbps or more.<sup id=\"rdp-ebb-cite_ref-SoNamed13_23-0\" class=\"reference\"><a href=\"#cite_note-SoNamed13-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KhoussiPerform19_24-0\" class=\"reference\"><a href=\"#cite_note-KhoussiPerform19-24\">[24]<\/a><\/sup>\n<\/p><p>When the Interest reaches a node or router with the requested data, it packages the content under the same name (i.e., the request name), signs it with the producer\u2019s signature, and returns it. For example, a request for \u201c\/google\/index.html\u201d brings back data under the same name \u201c\/google\/index.html\u201d that contains a payload with the actual data and the data producer\u2019s (i.e., Google) signature. This Data packet follows the reverse path taken by the Interest. Note that Interest or Data packets do not carry any host information or IP addresses; they are simply forwarded based on names (for Interest packets) or state in the routers (for Data packets). Since every NDN Data packet is signed, the router can store it locally in a cache to satisfy future requests.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Hierarchical_naming\">Hierarchical naming<\/span><\/h3>\n<p>There is no restriction on how content is named in NDN except they must be human-readable and hierarchical, and globally unique. The scientific communities develop the naming schemes as they see fit, and the uniqueness of names can be ensured by name registrars (similar to existing DNS Registrars).\n<\/p><p>The NDN design assumes hierarchically structured names, e.g., a genome sequence published by NCBI may have the name \u201c\/NCBI\/Human\/DNA\/Genome\/hg38\u201d, where \u201c\/\u201d indicates a separator between name components. The whole sequence may not fit in a single Data packet, so the segments (or chunks) of the sequence will have the names \u201c\/NCBI\/Human\/DNA\/Genome\/hg38\/{1..n}\u201c. Data that is routed and retrieved globally must have a globally unique name. This is achieved by creating a hierarchy of naming components, just like Domain Name System (DNS). In the example above, all sequences under NCBI will potentially reside under \u201c\/NCBI\u201d; \u201c\/NCBI\u201d is the name prefix that will be announced into the network. This hierarchical structure of names is useful both for applications and the network. For applications, it provides an opportunity to create structured, organized names. On the other hand, the network does not need to know all the possible content names, only a prefix. For example, \u201c\/NCBI\u201d is sufficient for forwarding.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data-centric_security\">Data-centric security<\/span><\/h3>\n<p>In NDN, security is built into the content. Each piece of data is signed by the data producer and is carried with the content. Data signatures are mandatory; on receiving the data, applications can decide if they trust the publisher or not. The signature, coupled with data publisher information, enables the determination of data provenance. NDN\u2019s data-centric security helps establish data provenance, e.g., users can verify content with names that begin with \u201c\/NCBI\u201d is digitally signed by NCBI\u2019s key.\n<\/p><p>NDN\u2019s data-centric security decouples content from its original publisher and enables in-network caching; it is no longer critical where the data comes from since the client can verify the authenticity of the data. Unsigned data is rejected either in the network or at the receiving client. The receiver can get content from anyone, such as a repository, a router cache, or a neighbor\u2014as well as the original publisher\u2014and verify that the data is authentic.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"In-network_caching\">In-network caching<\/span><\/h3>\n<p>Automatic in-network caching is enabled by naming data because a router can cache data packets in its content store to satisfy future requests. Unlike today\u2019s Internet, NDN routers can reuse the cached data packets since they have persistent names and the producer\u2019s signature. The cache (or Content Store) is an in-memory buffer that keeps packets temporarily for future requests. Data such as reference genomes can benefit from caching since caching the content near the user speeds up content delivery and reduces the load on the data servers. In addition to the CS, NDN supports persistent, disk-based repositories (repos).<sup id=\"rdp-ebb-cite_ref-ChenNDN14_25-0\" class=\"reference\"><a href=\"#cite_note-ChenNDN14-25\">[25]<\/a><\/sup> These storage devices can support caching at a larger scale and CDN-like functionality without additional application-layer engineering.\n<\/p><p>In our previous work with the climate science and high-energy physics communities, we saw that even though scientific data is large, a strong locality of reference exists. We found that for climate data, even a 1 GB cache in the network speeds up data distribution significantly.<sup id=\"rdp-ebb-cite_ref-ShannigrahiRequest17_17-1\" class=\"reference\"><a href=\"#cite_note-ShannigrahiRequest17-17\">[17]<\/a><\/sup> We observe similar patterns in the genomics community, where some of the reference genomes are very popular. These caches do not have to be at the core of the network. We anticipate most of the benefits will come from caching at the edge. For example, a large cache provisioned at the network gateway of a lab will benefit the scientists at that lab. In this case, the lab will provision and maintain their caches. If data is popular across many organizations, it is in the operators best interest to cache the data at the core since this will reduce latency and network traffic. Given that the price of data storage has gone down significantly (an 8 TB (8000 GB) hard-drive costs around $150, at the time of writing this paper), it does not significantly add to the operating costs of the labs. Additionally, new routers and switches are increasingly being shipped with storage, reducing the need for additional capital expenditure. Additionally, caching and cache maintenance is automated in NDN (it follows content popularity), eliminating the need to configure and maintain such storage.\n<\/p><p>Having introduced NDN in this section, we now enumerate the genomics data management problems and how NDN can solve them in the following section.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Genomics_cyberinfrastructure_challenges_and_solutions_using_NDN\">Genomics cyberinfrastructure challenges and solutions using NDN<\/span><\/h2>\n<p>The genomics community has made significant progress in recent decades. However, this progress has not been without challenges. A core challenge, like many other science domains, is data volume. Due to the low-cost sequencing instruments, the genomics community is rapidly approaching petascale data production at sequencing facilities housed in universities, research, and commercial centers. For example, the SRA repository at NCBI in Maryland, United States contains over 45 petabytes of high-throughput DNA sequence data, and there are other similar genomic data repositories around the world.<sup id=\"rdp-ebb-cite_ref-DDBJHome_26-0\" class=\"reference\"><a href=\"#cite_note-DDBJHome-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-EMBL-EBIHome_27-0\" class=\"reference\"><a href=\"#cite_note-EMBL-EBIHome-27\">[27]<\/a><\/sup> These data are complemented with <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a> (though not always present or complete) representing evolutionary relationships, biological sample sources, measurement techniques, and biological conditions.<sup id=\"rdp-ebb-cite_ref-NCBISequence20_12-1\" class=\"reference\"><a href=\"#cite_note-NCBISequence20-12\">[12]<\/a><\/sup>\n<\/p><p>Furthermore, while a large amount of data is accessible from large repositories such as the NCBI repository, a significant amount of genomics data resides in thousands of institutional repositories.<sup id=\"rdp-ebb-cite_ref-LatheGenomic08_28-0\" class=\"reference\"><a href=\"#cite_note-LatheGenomic08-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DankarTheDevel18_29-0\" class=\"reference\"><a href=\"#cite_note-DankarTheDevel18-29\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NationalData20_30-0\" class=\"reference\"><a href=\"#cite_note-NationalData20-30\">[30]<\/a><\/sup> The current (preferred) way to publish data is to upload it to a central repository, e.g., NCBI, which is time-consuming and often requires effort from the scientists. The massively distributed nature of the data makes the genomics community unique. In other scientific communities, such as high-energy physics (HEP), climate, and astronomy, only a few large scale repositories serve most of the data.<sup id=\"rdp-ebb-cite_ref-LHCStudyGroupDesign91_31-0\" class=\"reference\"><a href=\"#cite_note-LHCStudyGroupDesign91-31\">[31]<\/a><\/sup> For example, the Large Hadron Collider (LHC) produces most of the data for the HEP community at CERN, the telescopes (such as LSST and the to-be-built SKA) produces most of the data for astrophysics, and the supercomputers at various national labs produce climate simulation outputs.<sup id=\"rdp-ebb-cite_ref-TaylorCMIP5_10_32-0\" class=\"reference\"><a href=\"#cite_note-TaylorCMIP5_10-32\">[32]<\/a><\/sup>\n<\/p><p>Modern genomic data comes in the form of reference genomes with coordinate-based annotation files, \u201cdynamic\u201d measurements of genome output (e.g., RNA-seq, CHIP-seq), and individual genome resequencing data. Reference genomes are used by many researchers across the world who can benefit from efficient data delivery mechanisms. The dynamic functional genomics and resequencing genomics datasets are often larger in size and of more focused use. All data is typically retrieved using various contemporary technologies such as sneakernet<sup id=\"rdp-ebb-cite_ref-MunsonBulk13_33-0\" class=\"reference\"><a href=\"#cite_note-MunsonBulk13-33\">[33]<\/a><\/sup>, SCP\/FTP, IBM Aspera<sup id=\"rdp-ebb-cite_ref-IBMAspera_34-0\" class=\"reference\"><a href=\"#cite_note-IBMAspera-34\">[34]<\/a><\/sup>, Globus<sup id=\"rdp-ebb-cite_ref-UoCGlobus_35-0\" class=\"reference\"><a href=\"#cite_note-UoCGlobus-35\">[35]<\/a><\/sup>, and iRODS.<sup id=\"rdp-ebb-cite_ref-RajesekariRODS10_36-0\" class=\"reference\"><a href=\"#cite_note-RajesekariRODS10-36\">[36]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChiangImplem11_37-0\" class=\"reference\"><a href=\"#cite_note-ChiangImplem11-37\">[37]<\/a><\/sup> While reference data is often easier to locate and download, the dynamic and resequencing datasets often are not since they are strewn over geographically distributed institutional repositories. \n<\/p><p>Locating and retrieving data are not the only problems that the genomics community face. The rest of this section enumerates the cyberinfrastructure requirements of the genomics community, problems encountered due to the current point-to-point TCP\/IP-based model of the internet, and how NDN can solve these.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_massive_storage_problem\">The massive storage problem<\/span><\/h3>\n<p>The genomics community is producing more data than it is currently feasible to store locally.<sup id=\"rdp-ebb-cite_ref-StephensBigData15_13-1\" class=\"reference\"><a href=\"#cite_note-StephensBigData15-13\">[13]<\/a><\/sup> This phenomenon will accelerate as modern field-based or hand-held sequencers become more prevalent in individual research labs and commercial sequencing providers. Increasingly, valuable data is at risk of being lost, potentially forever. While the community must invest in storage capacity, the existing storage strategies need to be optimized, such as deduplication of popular datasets (e.g., the reference genomes). Moreover, popular datasets that are often reused must be available quickly and reliably to reduce the need for copying data.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"NDN-based_solution\">NDN-based solution<\/span><\/h4>\n<p>With NDN, data can come from anywhere, including in-network caches. Fast access to popular data reduces the need to download and store datasets locally. They can be quickly downloaded and deleted after the experiments. Multiple researchers in the same area can benefit from this approach since they no longer need to individually download datasets from NCBI, rather from a in-network cache that is automatically populated by the network. Further, the data downloaded from this cache can be verified publicly for provenance. Another solution is to push the computation to the data. This can be accomplished by adding a lambda (computational function) to the Interest name. The data source (e.g., a data producer or a dedicated service) interprets the lambda upon receiving the Interest and returns the computed results. Scientists don\u2019t have to download and store large datasets every time they need to run an experiment.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_data_discovery_problem\">The data discovery problem<\/span><\/h3>\n<p>Genomics data is currently published from central locations (e.g., NCBI, NASA). The challenges in data discovery come not only from the fact that one needs to know all the locations of these datasets but also to navigate different naming schemes and discovery mechanisms provided by the hosting entity. There are many community-supported efforts to define controlled vocabularies and <a href=\"https:\/\/www.limswiki.org\/index.php\/Ontology_(information_science)\" title=\"Ontology (information science)\" class=\"wiki-link\" data-key=\"52d0664bde4b458e81fbc128b911a4a6\">ontologies<\/a> to help describe data.<sup id=\"rdp-ebb-cite_ref-EilbeckTheSeq05_38-0\" class=\"reference\"><a href=\"#cite_note-EilbeckTheSeq05-38\">[38]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SchrimlDisease12_39-0\" class=\"reference\"><a href=\"#cite_note-SchrimlDisease12-39\">[39]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GOCGene15_40-0\" class=\"reference\"><a href=\"#cite_note-GOCGene15-40\">[40]<\/a><\/sup> These metadata then can be parsed, indexed, and organized for data discovery. A scientist, for example, can associate appropriate metadata with source data, resulting data, and data collections. Moreover, the application of metadata to data is non-uniform, non-standard, and often inconsistent, making them difficult to utilize for consistent naming or data discovery.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"NDN-based_solution_2\">NDN-based solution<\/span><\/h4>\n<p>NDN does not provide data discovery by itself. Once data is named consistently by a certain community or subcommunity, these names can be indexed by a separate application (see our previous work<sup id=\"rdp-ebb-cite_ref-FanManaging15_15-1\" class=\"reference\"><a href=\"#cite_note-FanManaging15-15\">[15]<\/a><\/sup> that provides name discovery functions and operated over NDN). Since name discovery in NDN is sufficient for data retrieval\u2014an application can request for this name\u2014no additional steps are necessary. Note that NDN only requires a hierarchical naming structure; how individual communities name their datasets (\/biology\/genome vs. \/genome\/biology) is up to them.<sup id=\"rdp-ebb-cite_ref-ShannigrahiWhat20_41-0\" class=\"reference\"><a href=\"#cite_note-ShannigrahiWhat20-41\">[41]<\/a><\/sup>\n<\/p><p>A distributed catalog<sup id=\"rdp-ebb-cite_ref-FanManaging15_15-2\" class=\"reference\"><a href=\"#cite_note-FanManaging15-15\">[15]<\/a><\/sup> that stores the content names is sufficient to provide efficient name discovery. Since an NDN-based catalog will only hold a community-specific set of names (not the actual data), the synchronization, update, and delete operations are lightweight.<sup id=\"rdp-ebb-cite_ref-ShannigrahiTheFut19_42-0\" class=\"reference\"><a href=\"#cite_note-ShannigrahiTheFut19-42\">[42]<\/a><\/sup> These names in these catalogs can be added, updated, and deleted as necessary. We refer the reader to our previous work for the details of how such a catalog can be created and maintained in NDN.<sup id=\"rdp-ebb-cite_ref-FanManaging15_15-3\" class=\"reference\"><a href=\"#cite_note-FanManaging15-15\">[15]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_fast_and_scalable_data_access_problem\">The fast and scalable data access problem<\/span><\/h3>\n<p>Currently, genomics data retrievals range from downloading a significant amount of data from a central data archive (e.g., NCBI) to downloading only the desired data that can be staged on local or cloud storage systems. For example, the researchers often need to retrieve genome reference data on demand for comparison. Downloading large amounts of datasets over long-distance internet links can be slow and error-prone. Further, the current internet model does not work very well over long distance links.<sup id=\"rdp-ebb-cite_ref-TierneyHigh10_43-0\" class=\"reference\"><a href=\"#cite_note-TierneyHigh10-43\">[43]<\/a><\/sup> Even with extremely high-speed links, it is particularly difficult to utilize all the available bandwidth.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"NDN-based_solution_3\">NDN-based solution<\/span><\/h4>\n<p>NDN provides access to data from \u201canywhere,\u201d including storage nodes, in-network caches, and any entities that might have the data. This property allows scientists to reuse already downloaded datasets that are nearby (e.g., dataset downloaded by another scientist in the same lab). Additionally, in NDN data follows the content popularity, as it is cached in the in-network devices automatically. The more popular content is, the higher the likelihood it would be cached nearby. All data is digitally signed, ensuring provenance is preserved.\n<\/p><p>Getting content fast and from nearby locations may be convenient to download data when needed and delete them when the computation is finished. For example, the reference human genome has been downloaded by us and our students hundreds of times in the last two decades. Secure and verifiable data downloaded on-demand will reduce the amount of storage needed.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_problem_of_minimizing_transfer_volume\">The problem of minimizing transfer volume<\/span><\/h3>\n<p>The massive data volume needed by genomic workflows can easily saturate institutional networks. For example, the size of sequence data (in FASTQ format) or processed sequence alignments (in binary sequence alignment map or BAM format) for one experiment can easily aggregate into terabytes. If stored in online repositories, these data might be downloaded many times by researchers extending existing studies, leading to high bandwidth usage. One solution is to subset the data and download only the necessary portion. However, several challenges remain. For example, if multiple copies of the file exist, the network\/application layers can not take advantage of that to pull different subsets in parallel.\n<\/p><p>However, depending on the size and type of the analysis being performed, subsetting of the data may not be appropriate. Currently, that means the scientist would be required to download all datasets (or staged at a remote site) before the computation can begin. However, instead of downloading large amounts of data, pushing computation to data might be much more lightweight.\n<\/p><p>For example, to determine if a scientific avenue (e.g., a large-scale experiment with millions of genomes) or dataset is worth pursuing, the scientists often run smaller-scale experiments for early signs of interesting properties. A key issue is determining the smallest number of records required to produce the same scientific result as the full dataset, and we previously point to a simple saturation point as determined by transcript detection.<sup id=\"rdp-ebb-cite_ref-MillsMoving19_44-0\" class=\"reference\"><a href=\"#cite_note-MillsMoving19-44\">[44]<\/a><\/sup> Once a saturation point has been reached, one could pause and examine the results. If there is an interesting signal, then there is nothing preventing the user from processing more sequence records. However, if there is no signal, one could drop the experiment and move on to other datasets. However, this method currently requires downloading the full experimental datasets and running computations against them.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"NDN-based_solution_4\">NDN-based solution<\/span><\/h4>\n<p>NDN supports subsetting the data at the source, and transferring only the necessary portions reduces bandwidth consumption. This is already possible through BAM file slicing to select data specific to genomic regions. With NDN, the request can carry the required subsetting parameters and allow the user\/applications to download only the part of the data required for computation. NDN can also parallelize subsetting in the event that multiple slices are needed, and data is replicated over multiple repositories. When subsetting is not appropriate, NDN is able to push computation to data by appending the computation to the Interest name (or adding them as the payload to the interest). The result comes back to the requester under the same name and is also cached for future use, reducing bandwidth usage. Furthermore, in some genomics workflows, caching of computation can reduce the load on the compute and servers (such as those hosted in NCBI or cloud platforms).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_secure_collaboration_problem\">The secure collaboration problem<\/span><\/h3>\n<p>Genomics data, especially unpublished or identifiable human data, can be very sensitive. Scientists often need to secure data due to privacy requirements, non-disclosure agreements, or legal restrictions (e.g., ). Without a security framework, securing data and enforcing permissions becomes difficult. Suitable data access methods with proper access control is therefore required for privacy and legal requirements. At the same time, scientific collaborations often need to share data between groups without violating security restrictions. Suitable frameworks must exist for utilizing open-source sequenced data for research, albeit with appropriately restricted access. The lack of an infrastructure that allows secure access to a large number of sequenced human genomes prevents population genetics researchers from identifying rare mutations or test hypotheses on analogous experiments which can lead to medical advancement. Encryption and data security models, along with proper access control is highly necessary as data breaches of protected data can lead to massive fines, forcing institutions to severely limit the scope of allowable controlled data access on local cyberinfrastructure.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"NDN-based_solution_5\">NDN-based solution<\/span><\/h4>\n<p>To support secure data sharing among collaborators, all data in NDN is digitally signed, providing data provenance. When privacy is needed, NDN allows encryption of content, facilitating secure collaborations. Furthermore, the verification of trust and associated operations (such as decryption) can be automated in NDN; this is called schematized trust.<sup id=\"rdp-ebb-cite_ref-YuSchema15_45-0\" class=\"reference\"><a href=\"#cite_note-YuSchema15-45\">[45]<\/a><\/sup> One example of schematized trust might be the following: a scientist attempting to decrypt a data packet starting with \u201c\/NCBI\u201d must also present a key that is signed by \u201c\/NCBI\u201d and begins with the \u201c\/NCBI\/scientistA\u201d. More complex, name-based trust schemes are also possible.\n<\/p><p>This section discussed NDN properties that can address data management and cyberinfrastructure challenges faced by the genomics community. In the following section, we present a pilot study that uses a current genomics workflow that demonstrates some of these improvements in a real-world scenario.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Method\">Method<\/span><\/h2>\n<p>To demonstrate how NDN can benefit genomics workflows, we integrated NDN with a current genomics workflow (GEMmaker) and deployed our integrated solution over a real, distributed NDN testbed.<sup id=\"rdp-ebb-cite_ref-ShannigrahiNamed15_16-1\" class=\"reference\"><a href=\"#cite_note-ShannigrahiNamed15-16\">[16]<\/a><\/sup> The experiment has multiple parts: 1) naming data in a way that is understood by NDN as well as acceptable to the genomics community (Figure 2); 2) publishing data into the testbed and making them discoverable to the users using a distributed catalog and a UI (Figure 3); 3) modifying GEMmaker to interact with the data published in the testbed; and 4) comparing the performance of the new integration to the existing workflow. The following sections describe these efforts in detail.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Ogle_FrontBigData2021_4.jpg\" class=\"image wiki-link\" data-key=\"33e4c534aa60143dc89703d2e2794f2e\"><img alt=\"Fig2 Ogle FrontBigData2021 4.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b3\/Fig2_Ogle_FrontBigData2021_4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 2.<\/b> Genome naming strategy for indexing in NDN. The NDN names were translated from the existing Pynome file naming scheme. Tokens above (surrounded by square brackets) indicate the location of taxonomic names and genome assembly names. Most of the names directly map to hierarchical NDN names. Depending on the use case, components can be added or removed. These names are starting points for all NDN based operations. They are also the only necessary component for an NDN network.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Ogle_FrontBigData2021_4.jpg\" class=\"image wiki-link\" data-key=\"d51bc9a68255a278ca41189d9d392739\"><img alt=\"Fig3 Ogle FrontBigData2021 4.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/84\/Fig3_Ogle_FrontBigData2021_4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 3.<\/b> The user interface (UI) for the NDN-based genomics catalog system. It demonstrates how a name based catalog can act as a central point to an NDN ecosystem. Once the user looks up the names, additional functionality such as data retrieval can be built on top these names.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"NDN_testbed\">NDN testbed<\/span><\/h3>\n<p>For this work, we utilized a geographically distributed six-node testbed that was deployed at Colorado State University, ESNet, and UCAR supercomputing center. The testbed had six high-performance nodes (each with 40 cores, 128 GB memory, and 50 TB storage) and connected over ESnet<sup id=\"rdp-ebb-cite_ref-ESnet_46-0\" class=\"reference\"><a href=\"#cite_note-ESnet-46\">[46]<\/a><\/sup> using 10 Gbps dedicated links. All nodes ran the latest version of Fedora, and the network stack was tuned for big-data transfers.<sup id=\"rdp-ebb-cite_ref-TierneyHigh10_43-1\" class=\"reference\"><a href=\"#cite_note-TierneyHigh10-43\">[43]<\/a><\/sup> Specifically, we tuned the network interfaces to increase the buffer size and used large ethernet frames (9000-byte jumbo frames). We also tuned the TCP stack according to the ESnet specification, including increasing read and write buffers, utilizing cubic and htcp congestion control algorithms. We also tuned the UDP stack to increase read\/write buffers as well as specifying CPU cores.<sup id=\"rdp-ebb-cite_ref-TierneyHigh10_43-2\" class=\"reference\"><a href=\"#cite_note-TierneyHigh10-43\">[43]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_naming_and_publication\">Data naming and publication<\/span><\/h3>\n<p>NDN recommends globally unique, hierarchical, and semantically meaningful names. This is a natural fit for the genomics community since they have established taxonomies dating as far back as 1773.\n<\/p><p>Note that while NDN requires names to be globally unique, there is no need for a global convention even among a particular community. For example, two names pointing to the same content\u2014 \/Biology\/Genome\/Homo\/Sapiens and \/NCBI\/Genome\/Homo\/Sapiens\u2014are perfectly acceptable. Each community is free to name their own content as they see fit. The uniqueness in these names come from the first component of the name (the prefix), which is \/Biology and \/NCBI, respectively. We anticipate each entity (e.g., an organization such as NCBI, a university, a community genome project) will have their own namespaces, possibly procured from commercial name registrars, the same way DNS namespaces are obtained today.\n<\/p><p>In the genomics community, very commonly used datasets can even be assigned their own namespaces. For example, a globally unique namespace (e.g., \/human\/genome) may be reserved for human genomes for convenience. However, that special namespace does not preclude an organization from publishing the same genomes from another namespace (e.g., \/NCBI\/human\/genome). While NDN operates on names, it does not interpret the semantic meaning of the name components at the network layer. For example, NCBI announces the \/NCBI prefix into the network that the NDN routers store. When an Interest named \/NCBI\/Genome\/Homo\/Sapiens arrives at the router, the router performs a longest prefix match on the name and matches the interface corresponding to \/NCBI. This way, the network is able to forward Interests and data but does not need to interpret the individual components.\n<\/p><p>It is true that a name component (e.g., \u201chg38\u201d) might have different meaning in different communities. It is the job of the application layer to interpret and use this components as they see fit. In our previous work, we built a catalog (an application) that mapped individual components to their semantic meaning (see Figure 3). Different communities will build different applications on top of NDN to understand the semantic meaning of a name.\n<\/p><p>The other problem is name assignment and reconciliation. In today\u2019s internet, ICCAN<sup id=\"rdp-ebb-cite_ref-ICANNWelcome_47-0\" class=\"reference\"><a href=\"#cite_note-ICANNWelcome-47\">[47]<\/a><\/sup> and the domain registrars play a significant role in assigning, maintaining and reconciling DNS namespaces (e.g., assigning google.com namespace to Google). In NDN, these organizations will continue to control and assign namespaces that are used over the internet.\n<\/p><p>In NDN, the data producer is responsible for publishing content under a name prefix (\/biology or \/NCBI). NCBI will acquire such namespaces from a name registrar. At that point, only NCBI is allowed to announce the \/NCBI prefix into the internet and publish content under that name prefix. The onus of updating namespaces is on the data publisher and the name registrars. Note that no such NDN name registrar exists today, but we expect similar organizations to exist in the future.\n<\/p><p>When a user looks up a content name (e.g., \/NCBI\/Genome\/Homo\/Sapiens) in a catalog and expresses an Interest, the Interest is forwarded by the network and eventually reaches the NCBI server that announced \/NCBI. Namespace reconciliation is not necessary for the network to function properly. Let\u2019s imagine NCBI is authorized to publish datasets under two namespaces\u2014\/Biology and \/NCBI\u2014and it publishes the human genome under \u201c\/Biology\/Genome\/Homo\/Sapiens\u201d and \u201c\/NCBI\/Genome\/Homo\/Sapiens\u201d. A user can utilize both names to retrieve content\u2014the network does not need to interpret the meanings of the names\u2014and the interpretation of name and content is up to the applications (and users) requesting and serving the datasets.\n<\/p><p>As part of the NSF SciDAS project<sup id=\"rdp-ebb-cite_ref-NSFSciDAS19_48-0\" class=\"reference\"><a href=\"#cite_note-NSFSciDAS19-48\">[48]<\/a><\/sup>, we store large amounts of whole genome reference and auxiliary data in a distributed data grid (iRODS). These whole genome references were initially retrieved from the Ensembl database<sup id=\"rdp-ebb-cite_ref-ZerbinoEnsembl18_49-0\" class=\"reference\"><a href=\"#cite_note-ZerbinoEnsembl18-49\">[49]<\/a><\/sup> using the Python-based Pynome<sup id=\"rdp-ebb-cite_ref-Pynome_50-0\" class=\"reference\"><a href=\"#cite_note-Pynome-50\">[50]<\/a><\/sup> package and consist of hundreds of Eukaryotic species. Pynome processes these data to provide indexed files which are not available on Ensembl and which are needed for common genomic applications performed by researchers around the world. These data are organized in an evolution-based, hierarchical manner which is an excellent naming convention for an NDN framework.\n<\/p><p>For this study, we used an NDN name translator that we created as part of our previous work<sup id=\"rdp-ebb-cite_ref-OlschanowskySupporting14_51-0\" class=\"reference\"><a href=\"#cite_note-OlschanowskySupporting14-51\">[51]<\/a><\/sup> to translate these existing names into NDN compatible names. Once translated, the content names became the unique reference for these datasets. Figure 2 shows the reference genome DNA sequence names created from the existing hierarchical naming convention. For example, one such name would look like \u201c\/Absidia\/glauca\/AG_v1\/fa\u201d. We could find that some of these names may or may not contain certain components, for example, infraspecific name in the above example. We then translated and imported these names into our NDN data management framework.<sup id=\"rdp-ebb-cite_ref-ShannigrahiNDN18_8-2\" class=\"reference\"><a href=\"#cite_note-ShannigrahiNDN18-8\">[8]<\/a><\/sup> All subsequent operations such as discovery, retrieval, and integration with workflows used these names.\n<\/p><p>After naming, we published these datasets under the derived names on the NDN testbed servers. In this pilot test, we used three nodes to publish the data. However, each testbed server published the same data under the same name, replicating the content. Depending on the client location, requests were routed to the closest replica. If one replica went down, nothing needed to change on the client\u2019s end\u2014the NDN network routed the request to a different replica. We then used this testbed setup in conjunction with GEMmaker to test NDN\u2019s usefulness. We discuss the results in the evaluation section.\n<\/p><p>The UI in Figure 3 provides an intuitive way to search for names that were published on the testbed. A user could create a query by selecting different components from the left-hand menu (e.g., by selecting \u201cAbsidia\u201d under genus). The user can start typing the full name, and the UI will provide a set of autocompleted names. Finally, the user could choose to view the entire name tree using the tree view. The catalog and the UI are necessary for dataset discovery; while NDN operates on names, it is more efficient and fast to discover names using a catalog. Once discovered, the names could be used for all subsequent operations. For example, once the names are known, the user can initiate retrieval or other operations, as discussed before. The API also provides a command-line interface (CLI) for name discovery, allowing the users to integrate the name discovery (and subsequent operations) with domain workflows.\n<\/p><p>Since NDN operates on names, data naming in NDN affects application and network behaviors. The way a piece of content is named has profound impacts on content discovery, routing of user requests, data retrieval, and security. Besides, the naming of individual pieces of content seriously affects how the NDN behaves. For brevity, we do not discuss those naming trade-offs here but point the reader to our recent work on content naming in NDN.<sup id=\"rdp-ebb-cite_ref-ShannigrahiWhat20_41-1\" class=\"reference\"><a href=\"#cite_note-ShannigrahiWhat20-41\">[41]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Integration_with_GEMmaker\">Integration with GEMmaker<\/span><\/h3>\n<p>GEMmaker<sup id=\"rdp-ebb-cite_ref-HadishSystems20_19-1\" class=\"reference\"><a href=\"#cite_note-HadishSystems20-19\">[19]<\/a><\/sup> is a workflow that takes as input a number of large RNA sequence (RNA-seq) data files and a whole genome reference file to quantify gene expression-levels underlying a specific set of experimental parameters. It produces a Gene Expression Matrix (GEM), which is a data structure that can be used to compare gene expression across the input RNA-seq samples. GEMs are commonly used in genomics and have been instrumental in several studies.<sup id=\"rdp-ebb-cite_ref-FicklinDiscov17_52-0\" class=\"reference\"><a href=\"#cite_note-FicklinDiscov17-52\">[52]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RocheCancer17_53-0\" class=\"reference\"><a href=\"#cite_note-RocheCancer17-53\">[53]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DunwoodieDiscov18_54-0\" class=\"reference\"><a href=\"#cite_note-DunwoodieDiscov18-54\">[54]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PoehlmanLink19_55-0\" class=\"reference\"><a href=\"#cite_note-PoehlmanLink19-55\">[55]<\/a><\/sup> GEMmaker consists of five major steps: \n<\/p>\n<dl><dd> 1. Read RNA-seq data files as input.<\/dd>\n<dd> 2. Trim low-quality portions of the sequences.<\/dd>\n<dd> 3. Map sequences to a reference genome. <\/dd>\n<dd> 4. Count levels of gene expression in each RNA-seq sample.<\/dd>\n<dd> 5. Merge the gene expression values of each sample into a Gene Expression Matrix. <\/dd><\/dl>\n<p>GEMmaker is regulated by the Nextflow<sup id=\"rdp-ebb-cite_ref-NextFlow_56-0\" class=\"reference\"><a href=\"#cite_note-NextFlow-56\">[56]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DiTommasoNext17_57-0\" class=\"reference\"><a href=\"#cite_note-DiTommasoNext17-57\">[57]<\/a><\/sup> workflow manager, which automates the fluid execution of each step in GEMmaker, which are defined as Nextflow processes. Nextflow has a variety of executors that enable the deployment of processes to a number of HPC and cloud environments, including Kubernetes (K8s).\n<\/p><p>Without NDN, the typical process a user follows to execute GEMmaker in a K8s environment is as follows. First, the user moves the whole genome reference files onto a Persistent Volume Claim (PVC), a K8s artifact that provides persistent storage to users. Nextflow is able to mount and unmount GEMmaker pods directly to this PVC, which stores all input, intermediate, and output data. When the user executes GEMmaker, Nextflow deploys pods that automatically retrieve the RNA-seq data directly from NCBI using Aspera or FTP services. One pod is submitted for each RNA sample in parallel, so ideally all RNA-seq data is downloaded simultaneously. This represents the bulk of data movement in GEMmaker. Once each sample of RNA-seq data is downloaded to the PVC, new pods are deployed to execute the next step of the workflow for each sample. Each step in GEMmaker produces a new intermediate file used as input for the next step, all of which is written to the PVC. Once GEMmaker completes, the resulting GEM can be downloaded from the PVC by the user.\n<\/p><p>Currently, pulling data from host-specific services is built into the GEMmaker workflow. For example, the NCBI SRA-toolkit can pull data from SRA but not all genome data repositories. With NDN, the process can be abstracted from the workflow logic as data is preloaded into the NDN testbed from any host, downloaded by name at step 1, reference data is pulled by name at step 3, and data is created and uploaded with a new name at step 5. In contrast to the typical execution of GEMmaker, with NDN, users need not retrieve the whole genome reference or the final GEM, and retrieval of RNA-seq data can occur independent of Aspera or FTP protocols, supporting a greater variety of data repositories or even locally created data (so long as a local publisher makes it available). Users need only provide the NDN names for moving data and accessing cloud APIs.\n<\/p><p>For GEMmaker integration, we created a NDN-capable program that pulled data from the NDN testbed using the SRA ID. We modified GEMmaker to replace the existing RNA-seq data retrieval step with this NDN retrieval program and to retrieve the whole genome references. These whole genome references are the same described previously that were generated by Pynome and cataloged in the testbed. For this small-scale testing, we added the NDN routes to the testbed machines manually. However, the NDN community provides multiple routing protocols<sup id=\"rdp-ebb-cite_ref-WangOptimal13_58-0\" class=\"reference\"><a href=\"#cite_note-WangOptimal13-58\">[58]<\/a><\/sup> that can automate the routing updates.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Performance_evaluation\">Performance evaluation<\/span><\/h3>\n<p>In order to understand how caching affects data distribution, we moved datasets of different sizes between the NDN testbed and Clemson University. The SRA sequences were published into the testbed, and then the workflow modified to download data over NDN. NDN-based download only needs to know the name of the content. The rest of the infrastructure is opaque to the user.\n<\/p><p>For these experiments, the standard NDN tools<sup id=\"rdp-ebb-cite_ref-AfanasyevNFD16_22-1\" class=\"reference\"><a href=\"#cite_note-AfanasyevNFD16-22\">[22]<\/a><\/sup> for publication and retrieval were used. Once the files were downloaded, they were utilized in the workflow. For the first experiment, we copied different sized files using existing IP based tools (wget). For comparison, we then utilized NDN-based standard tools (ndncatchunks and ndnputchunks) for the same files, with in-network caching disabled, followed by caching enabled. Each transfer experiment was repeated three times.\n<\/p><p>The experiments showed how NDN can improve content retrieval using in-network caching. Figure 4A shows a comparison of data download speed between NDN and HTTP. The first three bars represent three sequential downloads of a set of three SRA datasets from the NCBI repository using HTTP. The next three bars show three sequential downloads using NDN retrieval from the NDN testbed without caching. Since we manually staged the data, we knew that data source was approximately 1,500 miles away from the requester. Even then, the download performance was comparable with the HTTP performance. The real performance gain came from caching the datasets as seen in the last three bars where the first transfer was similar to HTTP and NDN without caching (two minutes) while the next two transfers only took around 20 seconds after caching kicked in. These results point toward a massive improvement opportunity since many genomics workflows download hundreds or even thousands of SRAs for a given experiment.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Ogle_FrontBigData2021_4.jpg\" class=\"image wiki-link\" data-key=\"31b33a263108c653c99187f89fd3fb4d\"><img alt=\"Fig4 Ogle FrontBigData2021 4.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/54\/Fig4_Ogle_FrontBigData2021_4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 4.<\/b> The effect of NDN caching on DNA dataset insertion into GEMmaker workflow. A set of three NCBI SRA <i>Arabidopsis thaliana<\/i> datasets (SRR5263229:167.9 Mb, SRR5263230:167.8 Mb, SRR5263231:167.9 Mb) was sequentially transferred three times (black bars [transfer I], gray bars [transfer II], white bars [transfer III]) for insertion into the GEMmaker workflow from the NDN testbed or from the SRA repository over the internet via HTTP. The y-axis in both panels shows the transfer time of all three datasets, from request to workflow accessibility, in seconds. Error bars represent standard error of the mean. (<b>A<\/b>) The aggregate transfer times are shown for the three sequential transfers with HTTP, NDN tested without caching, and NDN with caching at 655,360 packets. (<b>B<\/b>) The effect of varying cache size (packet number in parentheses) is shown. The x-axis shows the cache capacity in packet numbers.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Figure 4B shows how much in-network cache was needed to accomplish speedup using in-network caching and how the caching capacity affects the speedup. The x-axis of this figure shows the cache size in the number of NDN data packets (by default, each packet is 4,400 bytes). We find that at around 500 MB cache size, we start to see speed improvements.\n<\/p><p>We performed an additional experiment to better evaluate caching on data transfer in a cloud environment (Figure 5). In this caching experiment, gateway and endpoint containers were employed to determine the time it takes to download a SRA sequence dataset. The gateway container used NDN to pull the SRA sequence from a remote NDN repository and create the network cache. The endpoint container (with a cache size of 0) acted as the consumer and created a face to the gateway used to pull the data. Both the endpoint and the client were run on separate nodes to replicate real cloud scenarios. This experiment demonstrates how an NDN container can cache a dataset for use by any endpoint on the same network and indicates that an insufficiently sized content store on the gateway will prevent network caching, resulting in slower download times. This provides a basic example of how popular genomic sequences could be cached for use by multiple researchers working on the same network in the cloud.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Ogle_FrontBigData2021_4.jpg\" class=\"image wiki-link\" data-key=\"40d5ae22e83dd84de7703d665a8a90fa\"><img alt=\"Fig5 Ogle FrontBigData2021 4.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ec\/Fig5_Ogle_FrontBigData2021_4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 5.<\/b> The effect of NDN caching on SRA download times on a Kubernetes cluster. A single NCBI SRA <i>Homo sapiens<\/i> kidney RNAseq dataset (SRR5139395) was downloaded a total of nine times (3 transfer trials labeled I, II, III in triplicate) for each cache size (packet number in parentheses). The following cs_max sizes (packet number) were used: 65,536 (approx 500 MB), 262,144, 327,680, 393,216, and 458,752. The Y-axis shows the download time for the dataset into the endpoint pod in seconds. Error bars represent standard error of the mean.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The actual cache sizes in the real-world would depend on request patterns as well as file sizes\u2014we are currently working on quantifying this. In any case, it is certainly feasible to utilize NDN\u2019s caching ability for very popular datasets, such as the human reference genomes. Our previous work shows that in big-science communities, even a small amount of cache significantly speeds up delivery due to the temporal locality of requests.<sup id=\"rdp-ebb-cite_ref-ShannigrahiRequest17_17-2\" class=\"reference\"><a href=\"#cite_note-ShannigrahiRequest17-17\">[17]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Codebase\">Codebase<\/span><\/h2>\n<p>All code used for these experiments are publicly available and distributed under open source licences (Table 1). NFD is the Named Data Networking forwarder that works as a software router for Interests and Data packets. ndn-cxx is an NDN library that provides the necessary API for creating NDN based \u201capps\u201d. NDN catalog is an NDN based Name lookup system\u2014an application can send an Interest to the catalog and receive the name of a dataset. The application can then utilize the name for all subsequent operations. Two versions of the catalog exists, ndn-atmos is written in C++ while ndn-python-catalog is implemented in Python.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Ogle_FrontBigData2021_4.jpg\" class=\"image wiki-link\" data-key=\"baf4447cd1b464379855ece81b2e51a3\"><img alt=\"Tab1 Ogle FrontBigData2021 4.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d5\/Tab1_Ogle_FrontBigData2021_4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Table 1.<\/b> Software packages used in this experiment.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Nextflow is a general purpose workflow manager. GEMmaker is a genomics workflow adpated for Nextflow (see Section above for more details). For these experiments, we modified GEMmaker to request data over NDN instead of standard mechanisms (such as HTTP). This modified workflow is available under <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/mmcogle\/GEMmakerCam\" target=\"_blank\">this repository<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion_and_future_directions\">Discussion and future directions<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h3>\n<p>This work demonstrates the preliminary integration of NDN with genomics workflows. While the work shows the promise of NDN toward a simplified but capable cyberinfrastructure, several other aspects remain to be addressed before NDN can completely integrate with genomics workflows and cloud computing platforms. In this section we discuss the technical challenges as well as the economic considerations that are yet to be addressed.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Economic_considerations\">Economic considerations<\/span><\/h4>\n<p>NDN is a new internet architecture that operates differently that the current internet. Consequently, the users and the network operators need to consider the economic cost of moving to an NDN paradigm. This section outlines some of the economic considerations. There are two primary cost of moving to an NDN paradigm: the cost of upgrading current network equipment and the cost of storage if caching is desired.\n<\/p><p>As of writing this paper, NDN routers are predominately software-based. To utilize these software-based routers, the researcher needs to install NFD (an NDN packet forwarder) and NDN libraries on a machine (a server or even a laptop). All experiments in this paper were done on commodity hardware and did not require any additional capital investment. NDN is able to run as an overlay on top of the existing IP infrastructure or on Layer two circuits; in this work, we utilized NDN as an overlay on existing internet connectivity.\n<\/p><p>Any commodity hardware (servers or desktop) with storage (depends on the workflow requirement) and a few GB memory is capable of supporting NDN. Given the current low cost of storage (an 8 TB hard drive costs around $150) even the cost of moving to a dedicated server is low. However, workflows with large storage requirement will need some capital investment. To minimize this cost and provide an alternative, we are working on two storage access mechanisms. First, when installing new storage is feasible (e.g., cost of storage continues to fall and a petabyte of storage costs around $30,000 at the time of writing this paper) it might be convenient for large research organizations to install dedicated storage that holds and serves NDN data packets; this approach improves NDN\u2019s performance since objects are already packetized and signed by the data producers. The second approach is interfacing NDN with existing storage repositories such as HTTP and FTP servers. As NDN Interests come in, they are translated into appropriate system calls (e.g., POSIX, HTTP, or FTP) and the NDN Data packets are created on-demand. This approach is slower than the first approach but does not require any additional hardware or storage, reducing the deployment cost.\n<\/p><p>The benefits of NDN (caching, etc.) becomes apparent when more users utilize an NDN-based network. As the networking community moves toward testbeds and deployments such as NSF-funded FABRIC<sup id=\"rdp-ebb-cite_ref-FABRICHome_59-0\" class=\"reference\"><a href=\"#cite_note-FABRICHome-59\">[59]<\/a><\/sup> that incorporate NDN into their core design, the research labs and institutes connected to these networks would be able to take advantage of those infrastructures. Additionally, connecting a software NDN router to these networks and testbeds are often free (assuming an institute is already paying for internet access). These networks will create and deploy large scale in-network caches near the users and in the internet core as users continue to request data from. As users exchange data over these networks, data will be automatically cached and served from in-network caches. In the future, when ISPs deploy NDN routers, the researchers will be able to take advantage of in-network caching without added cost. However, we expect large scale ISP deployment of NDN to take a few more years.\n<\/p><p>The other cost is the learning and integration cost with existing workflows. This is not trivial; NDN requires careful naming considerations, aligning workflows with a name base network, and data publication. To make this process straightforward (Figure 6), we are working on containerizing several individual pieces. We hope that containerizing NDN, data transfer pods, and other components will allow researchers to simply mix-and-match different containers and integrate them with workflows without resorting to complex configuration and integration efforts. Having discussed the economic considerations, we now discuss the technical challenges that remain.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Ogle_FrontBigData2021_4.jpg\" class=\"image wiki-link\" data-key=\"aadbf4dbc3dbe4121f750ecfd3fa51e6\"><img alt=\"Fig6 Ogle FrontBigData2021 4.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/5a\/Fig6_Ogle_FrontBigData2021_4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 6.<\/b> Nextflow managed GEMmaker workflow on a Kubernetes Cluster.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Future_directions\">Future directions<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Software_performance\">Software performance<\/span><\/h4>\n<p>While our work shows some attractive properties of NDN for the genomics community, there are well-known shortcomings of NDN. For example, the forwarder we used (NFD) was single-threaded and therefore its throughput is low. This has been recently addressed by a new forwarder (ndn-dpdk) that can perform forwarding at 100 Gbps. We are currently working on integrating the genomics workflow with NDN-DPDK. We hope to demonstrate further improvements as the protocol and software stack continues to mature.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Accessing_distributed_data_over_NDN.3D\">Accessing distributed data over NDN=<\/span><\/h4>\n<p>Genomics data generation is highly distributed as data is generated in academic institutions, research labs, and the industry. While the HTDS datasets are eventually converted into in FASTQ format<sup id=\"rdp-ebb-cite_ref-CockTheSanger10_60-0\" class=\"reference\"><a href=\"#cite_note-CockTheSanger10-60\">[60]<\/a><\/sup>, the storage and downstream analysis data formats are highly heterogeneous with different naming systems, different storage techniques, and inconsistent and non-standard use of metadata. Seamlessly accessing these diverse datasets is an enormous challenge. Additionally, multiple types of repositories exist with different scopes: national-level (e.g., NCBI, EBI, and DDBJ) with large scale sequencing data for many organisms, community-supported repositories focused on a species or small clade of organisms, and single investigator web sites containing <i>ad hoc<\/i> datasets. Storing these datasets in different repositories that are hosted under various domain names is unlikely to scale very well for two primary reasons. First, different hosting entities utilize different schemes for data access APIs (e.g., URLs), making it necessary to understand and parse various naming schemes. Second, it is hard to find and catalog each institutional repository.\n<\/p><p>NDN provides a scalable approach to publish and retrieve immutable content using their names in a location-agnostic fashion. NDN uses the content names for routing and request forwarding, ensuring all publicly available data can be access directly without the need for frequent housekeeping. For example, currently moving a repository under a new domain name requires a large amount of housekeeping, such as renaming the data under a new URL or linking new and old data. With NDN, the physical location of the data has no bearing on how they are named or distributed. When data is replicated, NDN brings the requests to the \u201cnearest\u201d data replica. \u201cNearest\u201d in NDN can be defined as the physically nearest replica, the most performant replica, or a combination of these (and other) factors.\n<\/p><p>However, several unexplored challenges exist on applying NDN to distributed data. First, NDN operates on names. Finding these names requires the service of a third-party software or name provider. A catalog that enumerates all names under a namespace (as we discuss before) can provide this service. However, it is not yet obvious who would be responsible for running and updating the authoritative versions of these catalogs. When data is replicated, we also need to address the issue of data consistency across multiple repositories. This is still an active research direction that requires considerable attention.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Distributed_repositories_over_NDN\">Distributed repositories over NDN<\/span><\/h4>\n<p>A single centralized repository can become a bottleneck when subjected to a large number of queries or download requests. Moreover, it becomes a single point of failure and introduces increased latency for distant clients. NDN makes it easier to create distributed repositories since they no longer need to be tracked by their IP addresses. For example, in this work, we created a federation of three geographically distributed repositories. These repositories had to announce the prefix they intended to serve (e.g., \u201c\/genome\u201d). Even when one or more repositories go down or become unreachable, no additional maintenance is necessary; NDN routes the request to the available repositories as long as at least one repository is reachable. Similarly, when new repositories are added, the process is completely transparent to the user and does not require any action from the network administrator. However, several aspects remain to be addressed: how to allow data producers to publish long-term data efficiently, how to replicate datasets across repositories (partially or completely), and how to retrieve content most efficiently.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Publication_of_more_genomics_datasets_and_metadata_into_the_NDN_testbed\">Publication of more genomics datasets and metadata into the NDN testbed<\/span><\/h4>\n<p>There are currently hundreds of reference genomes in the NDN testbed, and we are working on updating Pynome to include more current genome builds, genomes from services other than Ensembl, and support for popular sequence aligner programs (e.g., StAR, Salmon, Kallisto, and Hisat2). We are also working on loading the metadata and SRA RNA-seq files at scale into the NDN testbed. By studying the usage logs of our integration, we will better understand the benefit NDN brings to genomics workflows. Further, the convergence of searchable metadata from multiple data repositories published in the same NDN testbed will allow for a common search and access point for genomic data.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Integration_with_Docker\">Integration with Docker<\/span><\/h4>\n<p>For further simplification of workflows, we have created a Docker container with NDN tools, forwarder, and application libraries built-in. The resulting container is fairly lightweight. We plan to publish the image to a public repository where scientists can download and utilize the docker build \u201cas-is\u201d. These images can be deployed to a variety of cloud platforms without modifications, further simplifying the NDN access to genomics workflows.\n<\/p><p>GEMmaker is able to run Nextflow processes inside specified containers. By adding a container that is configured with NDN to the GEMmaker directory, scripts can run inside the NDN container during a normal GEMmaker workflow. The GEMmaker workflow can then use the ndn-tools to download the SRA sequences, both from ndn-only or NDN-interfaced existing repositories. This method also provides the opportunity for decreased data retrieval time due to NDN in-network caching and allows GEMMaker to benefit from all the NDN features we described earlier.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Integration_With_Kubernetes\">Integration With Kubernetes<\/span><\/h4>\n<p>The genomics community is moving toward a cloud-based model. Container orchestration platforms (such as Kubernetes) are more commonly being used in favor of traditional HPC clusters. We believe that enabling users to easily move data from an NDN network to a Kubernetes cluster is imperative for the widespread adoption of this use case.\n<\/p><p>To achieve this goal, we are engineering the integration of NDN with a Data Transfer Pod (DTP) that runs as a deployment of different containers that enable users to read and write data to a Kubernetes cluster. A DTP is a configurable collection of containers, each representing a different data transfer protocol\/interface. A DTP uses the officially maintained images of each protocol\/interface, removing the need for integration, aside from adding the container to the DTP deployment, which is an almost identical process for each protocol\/interface. This makes adding new protocols very simple, as there is no need to build a custom image for each protocol\/interface, as long as an image already exists. Almost all interfaces\/protocols used by the community have officially maintained images associated with them.\n<\/p><p>The DTP tool will allow Kubernetes users to easily access data stored on NDN-based repositories. Each DTP container provides the client with a mechanism to utilize a different data transfer method (e.g., NDN, Aspera, Globus, S3). Using a DTP aims to address the need for simple access to data from a variety of sources. A DTP is not coupled with any particular workflow, so users will be able to pull or push NDN data as a pre- or post-processing step of their workflow, without modifying the workflow itself. The DTP can also be used to pull data from other sources if it is not present in an NDN framework.\n<\/p><p>We are modifying the existing GEMmaker workflow to use an NDN via the DTP that queries the NDN framework for the required reference genome files and SRA datasets. If the SRA dataset file exists in the NDN framework, the DTP will pull the data onto a Kubernetes persistent volume claim (PVC). For example, we are adapting NDN to work with the Pacific Research Platform (PRP)<sup id=\"rdp-ebb-cite_ref-SmarrThePac18_61-0\" class=\"reference\"><a href=\"#cite_note-SmarrThePac18-61\">[61]<\/a><\/sup> Nautilus Kubernetes cluster (6). If the user knows the content name (or metadata) but content does not exist in NDN format, the DTP will pull the dataset from SRA with Aspera and publish it in the NDN testbed and then pull into PRP. Once published, the dataset will now exist in the NDN testbed and benefit from NDN attributes, including caching. Once the DTP completes its job, the GEMmaker workflow will function in the same way it does now so no new code needs to be written. We are also developing a cache retention policy to allow the SRA files to evaporate if they are not accessed after a certain period of time.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Limitations\">Limitations<\/span><\/h3>\n<p>The NDN prototype we used (NFD) and other components we used (catalogs and repo) are research prototypes. The performance and scalability of these prototypes are being improved by the NDN community. Additionally, utilization of NDN containers on PRP has not been explored before; we are working on optimizing both the containers and their interactions with the cloud platforms. We are also working on better understanding the caching and storage needs of the genomics community by looking at real-world request patterns and object sizes associated with them.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>In this paper, we enumerate the cyberinfrastruture challenges faced by the genomics community. We discuss NDN, a novel but well-researched future internet architecture that can address these challenges at the network layer. We present our efforts in integrating NDN with a genomics workflow, GEMmaker. We describe the creation of an NDN-complaint naming scheme that is also acceptable to the genomics community. We find that genomics names are already hierarchical and easily translated into NDN names. We publish the actual datasets into an NDN testbed and show that NDN can serve data from anywhere, simplifying data management. Finally, through the integration with GEMmaker, we show NDN\u2019s in-network caching can speed up data retrieval and insertion into the workflow by six times.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>FF, SS, and SF are the PIs who designed the study, prepared and edited the manuscript, and provided guidance to the students and programmers. CO, DR, CM, TB, and RP are the students and programmers who performed GEMmaker integration, deployed the code and prototype on Pacific Research Platform and evaluated it. They also helped writing the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_availability_statement\">Data availability statement<\/span><\/h3>\n<p>The datasets presented in this study can be found in online repositories. The names of the repository\/repositories and accession number(s) can be found in the article.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work was supported by National Science Foundation Award #1659300 \u201cCC*Data: National Cyberinfrastructure for Scientific Data Analysis at Scale (SciDAS),\u201d National Science Foundation Award #2019012 \u201cCC* Integration-Large: N-DISE: NDN for Data Intensive Science Experiments,\u201d and Tennessee Tech Faculty Research Grant. National Science Foundation Award #2019163 CC* Integration-Small: Error Free File Transfer for Big Science.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-CinquiniTheEarth14-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CinquiniTheEarth14_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cinquini, L.; Chrichton, D.; Mattmann, C. et al. (2014). \"The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data\". <i>Future Generation Computer Systems<\/i> <b>36<\/b>: 400\u201317. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.future.2013.07.002\" target=\"_blank\">10.1016\/j.future.2013.07.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Earth+System+Grid+Federation%3A+An+open+infrastructure+for+access+to+distributed+geospatial+data&rft.jtitle=Future+Generation+Computer+Systems&rft.aulast=Cinquini%2C+L.%3B+Chrichton%2C+D.%3B+Mattmann%2C+C.+et+al.&rft.au=Cinquini%2C+L.%3B+Chrichton%2C+D.%3B+Mattmann%2C+C.+et+al.&rft.date=2014&rft.volume=36&rft.pages=400%E2%80%9317&rft_id=info:doi\/10.1016%2Fj.future.2013.07.002&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AtlasCollabTheAtlas08-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AtlasCollabTheAtlas08_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">ATLAS Collaboration; Aad, G.; Abat, E. et al. (2008). \"The ATLAS Experiment at the CERN Large Hadron Collider\". <i>Journal of Instrumentation<\/i> <b>3<\/b>: S08003. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1088%2F1748-0221%2F3%2F08%2FS08003\" target=\"_blank\">10.1088\/1748-0221\/3\/08\/S08003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+ATLAS+Experiment+at+the+CERN+Large+Hadron+Collider&rft.jtitle=Journal+of+Instrumentation&rft.aulast=ATLAS+Collaboration%3B+Aad%2C+G.%3B+Abat%2C+E.+et+al.&rft.au=ATLAS+Collaboration%3B+Aad%2C+G.%3B+Abat%2C+E.+et+al.&rft.date=2008&rft.volume=3&rft.pages=S08003&rft_id=info:doi\/10.1088%2F1748-0221%2F3%2F08%2FS08003&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DewdneyTheSquare09-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DewdneyTheSquare09_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dewdney, P.E.; Hall, P.J.; Schillizzi, R.T. et al. (2009). \"The Square Kilometre Array\". <i>Proceedings of the IEEE<\/i> <b>97<\/b> (8): 1482-1496. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FJPROC.2009.2021005\" target=\"_blank\">10.1109\/JPROC.2009.2021005<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Square+Kilometre+Array&rft.jtitle=Proceedings+of+the+IEEE&rft.aulast=Dewdney%2C+P.E.%3B+Hall%2C+P.J.%3B+Schillizzi%2C+R.T.+et+al.&rft.au=Dewdney%2C+P.E.%3B+Hall%2C+P.J.%3B+Schillizzi%2C+R.T.+et+al.&rft.date=2009&rft.volume=97&rft.issue=8&rft.pages=1482-1496&rft_id=info:doi\/10.1109%2FJPROC.2009.2021005&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbateLarge12-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbateLarge12_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">LSST Dark Energy Science Collaboration; Abate, A.; Aldering, G. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/arxiv.org\/abs\/1211.0310v1\" target=\"_blank\">\"Large Synoptic Survey Telescope Dark Energy Science Collaboration\"<\/a>. <i>arXiv<\/i>: 1\u2013133<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/arxiv.org\/abs\/1211.0310v1\" target=\"_blank\">https:\/\/arxiv.org\/abs\/1211.0310v1<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Large+Synoptic+Survey+Telescope+Dark+Energy+Science+Collaboration&rft.jtitle=arXiv&rft.aulast=LSST+Dark+Energy+Science+Collaboration%3B+Abate%2C+A.%3B+Aldering%2C+G.+et+al.&rft.au=LSST+Dark+Energy+Science+Collaboration%3B+Abate%2C+A.%3B+Aldering%2C+G.+et+al.&rft.date=2012&rft.pages=1%E2%80%93133&rft_id=https%3A%2F%2Farxiv.org%2Fabs%2F1211.0310v1&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SayersData20-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SayersData20_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-SayersData20_5-1\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sayers, E.W.; Beck, J.; Brister, J.R. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6943063\" target=\"_blank\">\"Database resources of the National Center for Biotechnology Information\"<\/a>. <i>Nucleic Acids Research<\/i> (D1): D9\u2013D16. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkz899\" target=\"_blank\">10.1093\/nar\/gkz899<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6943063\/\" target=\"_blank\">PMC6943063<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31602479\" target=\"_blank\">31602479<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6943063\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6943063<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Database+resources+of+the+National+Center+for+Biotechnology+Information&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Sayers%2C+E.W.%3B+Beck%2C+J.%3B+Brister%2C+J.R.+et+al.&rft.au=Sayers%2C+E.W.%3B+Beck%2C+J.%3B+Brister%2C+J.R.+et+al.&rft.date=2020&rft.issue=D1&rft.pages=D9%E2%80%93D16&rft_id=info:doi\/10.1093%2Fnar%2Fgkz899&rft_id=info:pmc\/PMC6943063&rft_id=info:pmid\/31602479&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6943063&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TsuchiyaBigData12-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TsuchiyaBigData12_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tsuchiya, S.; Sakamoto, Y.; Tsuchimoto, Y. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.fujitsu.com\/downloads\/MAG\/vol48-2\/paper09.pdf\" target=\"_blank\">\"Big Data Processing in Cloud Environments\"<\/a> (PDF). <i>Fujitsu Scientific & Technical Journal<\/i> <b>48<\/b> (2): 159\u201368<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.fujitsu.com\/downloads\/MAG\/vol48-2\/paper09.pdf\" target=\"_blank\">https:\/\/www.fujitsu.com\/downloads\/MAG\/vol48-2\/paper09.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Big+Data+Processing+in+Cloud+Environments&rft.jtitle=Fujitsu+Scientific+%26+Technical+Journal&rft.aulast=Tsuchiya%2C+S.%3B+Sakamoto%2C+Y.%3B+Tsuchimoto%2C+Y.+et+al.&rft.au=Tsuchiya%2C+S.%3B+Sakamoto%2C+Y.%3B+Tsuchimoto%2C+Y.+et+al.&rft.date=2012&rft.volume=48&rft.issue=2&rft.pages=159%E2%80%9368&rft_id=https%3A%2F%2Fwww.fujitsu.com%2Fdownloads%2FMAG%2Fvol48-2%2Fpaper09.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LuoBigData16-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LuoBigData16_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Luo, J.; Wu, M.; Gopukumar, D. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720168\" target=\"_blank\">\"Big Data Application in Biomedical Research and Health Care: A Literature Review\"<\/a>. <i>Biomedical Informatics Insights<\/i> <b>8<\/b>: 1\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4137%2FBII.S31559\" target=\"_blank\">10.4137\/BII.S31559<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4720168\/\" target=\"_blank\">PMC4720168<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26843812\" target=\"_blank\">26843812<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720168\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720168<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Big+Data+Application+in+Biomedical+Research+and+Health+Care%3A+A+Literature+Review&rft.jtitle=Biomedical+Informatics+Insights&rft.aulast=Luo%2C+J.%3B+Wu%2C+M.%3B+Gopukumar%2C+D.+et+al.&rft.au=Luo%2C+J.%3B+Wu%2C+M.%3B+Gopukumar%2C+D.+et+al.&rft.date=2016&rft.volume=8&rft.pages=1%E2%80%9310&rft_id=info:doi\/10.4137%2FBII.S31559&rft_id=info:pmc\/PMC4720168&rft_id=info:pmid\/26843812&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4720168&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShannigrahiNDN18-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShannigrahiNDN18_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShannigrahiNDN18_8-1\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-ShannigrahiNDN18_8-2\">8.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shannigrahi, S.; Fan, C.; Papadopoulos, C. et al. (2018). \"NDN-SCI for managing large scale genomics data\". <i>Proceedings of the 5th ACM Conference on Information-Centric Networking<\/i>: 204\u201305. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3267955.3269022\" target=\"_blank\">10.1145\/3267955.3269022<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=NDN-SCI+for+managing+large+scale+genomics+data&rft.jtitle=Proceedings+of+the+5th+ACM+Conference+on+Information-Centric+Networking&rft.aulast=Shannigrahi%2C+S.%3B+Fan%2C+C.%3B+Papadopoulos%2C+C.+et+al.&rft.au=Shannigrahi%2C+S.%3B+Fan%2C+C.%3B+Papadopoulos%2C+C.+et+al.&rft.date=2018&rft.pages=204%E2%80%9305&rft_id=info:doi\/10.1145%2F3267955.3269022&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MccombieNext19-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MccombieNext19_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McCombie, W.R.; McPherson, J.D.; Mardis, E.R. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720168\" target=\"_blank\">\"Next-Generation Sequencing Technologies\"<\/a>. <i>Cold Spring Harbor Perspectives in Medicine<\/i> <b>9<\/b> (11): 1\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4137%2FBII.S31559\" target=\"_blank\">10.4137\/BII.S31559<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4720168\/\" target=\"_blank\">PMC4720168<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26843812\" target=\"_blank\">26843812<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720168\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720168<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Next-Generation+Sequencing+Technologies&rft.jtitle=Cold+Spring+Harbor+Perspectives+in+Medicine&rft.aulast=McCombie%2C+W.R.%3B+McPherson%2C+J.D.%3B+Mardis%2C+E.R.&rft.au=McCombie%2C+W.R.%3B+McPherson%2C+J.D.%3B+Mardis%2C+E.R.&rft.date=2016&rft.volume=9&rft.issue=11&rft.pages=1%E2%80%9310&rft_id=info:doi\/10.4137%2FBII.S31559&rft_id=info:pmc\/PMC4720168&rft_id=info:pmid\/26843812&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4720168&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NIHTheCost20-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NIHTheCost20_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Human Genome Research Institute (2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.genome.gov\/about-genomics\/fact-sheets\/Sequencing-Human-Genome-cost\" target=\"_blank\">\"The Cost of Sequencing a Human Genome\"<\/a>. National Institutes of Health<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.genome.gov\/about-genomics\/fact-sheets\/Sequencing-Human-Genome-cost\" target=\"_blank\">https:\/\/www.genome.gov\/about-genomics\/fact-sheets\/Sequencing-Human-Genome-cost<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 16 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Cost+of+Sequencing+a+Human+Genome&rft.atitle=&rft.aulast=National+Human+Genome+Research+Institute&rft.au=National+Human+Genome+Research+Institute&rft.date=2020&rft.pub=National+Institutes+of+Health&rft_id=https%3A%2F%2Fwww.genome.gov%2Fabout-genomics%2Ffact-sheets%2FSequencing-Human-Genome-cost&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Lowy-GallegoVariant19-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Lowy-GallegoVariant19_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lowy-Gallego, E.; Fairley, S.; Zheng-Bradley, X. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7059836\" target=\"_blank\">\"Variant calling on the GRCh38 assembly with the data from phase three of the 1000 Genomes Project\"<\/a>. <i>Wellcome Open Research<\/i> <b>4<\/b>: 50. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.12688%2Fwellcomeopenres.15126.2\" target=\"_blank\">10.12688\/wellcomeopenres.15126.2<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7059836\/\" target=\"_blank\">PMC7059836<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32175479\" target=\"_blank\">32175479<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7059836\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7059836<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Variant+calling+on+the+GRCh38+assembly+with+the+data+from+phase+three+of+the+1000+Genomes+Project&rft.jtitle=Wellcome+Open+Research&rft.aulast=Lowy-Gallego%2C+E.%3B+Fairley%2C+S.%3B+Zheng-Bradley%2C+X.+et+al.&rft.au=Lowy-Gallego%2C+E.%3B+Fairley%2C+S.%3B+Zheng-Bradley%2C+X.+et+al.&rft.date=2019&rft.volume=4&rft.pages=50&rft_id=info:doi\/10.12688%2Fwellcomeopenres.15126.2&rft_id=info:pmc\/PMC7059836&rft_id=info:pmid\/32175479&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7059836&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NCBISequence20-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NCBISequence20_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-NCBISequence20_12-1\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">NCBI (2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/trace.ncbi.nlm.nih.gov\/Traces\/sra\/\" target=\"_blank\">\"Sequence Read Archive\"<\/a>. NCBI<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/trace.ncbi.nlm.nih.gov\/Traces\/sra\/\" target=\"_blank\">https:\/\/trace.ncbi.nlm.nih.gov\/Traces\/sra\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 February 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Sequence+Read+Archive&rft.atitle=&rft.aulast=NCBI&rft.au=NCBI&rft.date=2020&rft.pub=NCBI&rft_id=https%3A%2F%2Ftrace.ncbi.nlm.nih.gov%2FTraces%2Fsra%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StephensBigData15-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-StephensBigData15_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-StephensBigData15_13-1\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stephens, Z.D.; Lee, S.Y.; Faghri, F. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4494865\" target=\"_blank\">\"Big Data: Astronomical or Genomical?\"<\/a>. <i>PLoS Biology<\/i> <b>13<\/b> (7): e1002195. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pbio.1002195\" target=\"_blank\">10.1371\/journal.pbio.1002195<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4494865\/\" target=\"_blank\">PMC4494865<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26151137\" target=\"_blank\">26151137<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4494865\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4494865<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Big+Data%3A+Astronomical+or+Genomical%3F&rft.jtitle=PLoS+Biology&rft.aulast=Stephens%2C+Z.D.%3B+Lee%2C+S.Y.%3B+Faghri%2C+F.+et+al.&rft.au=Stephens%2C+Z.D.%3B+Lee%2C+S.Y.%3B+Faghri%2C+F.+et+al.&rft.date=2015&rft.volume=13&rft.issue=7&rft.pages=e1002195&rft_id=info:doi\/10.1371%2Fjournal.pbio.1002195&rft_id=info:pmc\/PMC4494865&rft_id=info:pmid\/26151137&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4494865&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OlschanowskySupport14-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OlschanowskySupport14_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Olschanowsky, C.; Shannigrahi, S. Papadopolous, C. et al. (2014). \"Supporting climate research using named data networking\". <i>Proceedings of the IEEE 20th International Workshop on Local & Metropolitan Area Networks<\/i>: 1\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FLANMAN.2014.7028640\" target=\"_blank\">10.1109\/LANMAN.2014.7028640<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Supporting+climate+research+using+named+data+networking&rft.jtitle=Proceedings+of+the+IEEE+20th+International+Workshop+on+Local+%26+Metropolitan+Area+Networks&rft.aulast=Olschanowsky%2C+C.%3B+Shannigrahi%2C+S.+Papadopolous%2C+C.+et+al.&rft.au=Olschanowsky%2C+C.%3B+Shannigrahi%2C+S.+Papadopolous%2C+C.+et+al.&rft.date=2014&rft.pages=1%E2%80%936&rft_id=info:doi\/10.1109%2FLANMAN.2014.7028640&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FanManaging15-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FanManaging15_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-FanManaging15_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-FanManaging15_15-2\">15.2<\/a><\/sup> <sup><a href=\"#cite_ref-FanManaging15_15-3\">15.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fan, C.; Shannigrahi, S.; DiBenedetto, S. et al. (2015). \"Managing scientific data with named data networking\". <i>Proceedings of the Fifth International Workshop on Network-Aware Data Management<\/i>: 1\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2832099.2832100\" target=\"_blank\">10.1145\/2832099.2832100<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Managing+scientific+data+with+named+data+networking&rft.jtitle=Proceedings+of+the+Fifth+International+Workshop+on+Network-Aware+Data+Management&rft.aulast=Fan%2C+C.%3B+Shannigrahi%2C+S.%3B+DiBenedetto%2C+S.+et+al.&rft.au=Fan%2C+C.%3B+Shannigrahi%2C+S.%3B+DiBenedetto%2C+S.+et+al.&rft.date=2015&rft.pages=1%E2%80%937&rft_id=info:doi\/10.1145%2F2832099.2832100&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShannigrahiNamed15-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShannigrahiNamed15_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShannigrahiNamed15_16-1\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shannigrahi, S.; Papadopolous, C.; Yeh, E. et al. (2015). \"Named Data Networking in Climate Research and HEP Applications\". <i>Journal of Physics: Conference Series<\/i> <b>664<\/b>: 052033. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1088%2F1742-6596%2F664%2F5%2F052033\" target=\"_blank\">10.1088\/1742-6596\/664\/5\/052033<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Named+Data+Networking+in+Climate+Research+and+HEP+Applications&rft.jtitle=Journal+of+Physics%3A+Conference+Series&rft.aulast=Shannigrahi%2C+S.%3B+Papadopolous%2C+C.%3B+Yeh%2C+E.+et+al.&rft.au=Shannigrahi%2C+S.%3B+Papadopolous%2C+C.%3B+Yeh%2C+E.+et+al.&rft.date=2015&rft.volume=664&rft.pages=052033&rft_id=info:doi\/10.1088%2F1742-6596%2F664%2F5%2F052033&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShannigrahiRequest17-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShannigrahiRequest17_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShannigrahiRequest17_17-1\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-ShannigrahiRequest17_17-2\">17.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shannigrahi, S.; Fan, C.; Papadopolous, C. (2017). \"Request aggregation, caching, and forwarding strategies for improving large climate data distribution with NDN: a case study\". <i>Proceedings of the 4th ACM Conference on Information-Centric Networking<\/i>: 54\u201365. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3125719.3125722\" target=\"_blank\">10.1145\/3125719.3125722<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Request+aggregation%2C+caching%2C+and+forwarding+strategies+for+improving+large+climate+data+distribution+with+NDN%3A+a+case+study&rft.jtitle=Proceedings+of+the+4th+ACM+Conference+on+Information-Centric+Networking&rft.aulast=Shannigrahi%2C+S.%3B+Fan%2C+C.%3B+Papadopolous%2C+C.&rft.au=Shannigrahi%2C+S.%3B+Fan%2C+C.%3B+Papadopolous%2C+C.&rft.date=2017&rft.pages=54%E2%80%9365&rft_id=info:doi\/10.1145%2F3125719.3125722&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShannigrahiNamed18-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShannigrahiNamed18_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shannigrahi, S.; Fan, C.; Papadopolous, C. et al. (2018). \"Named Data Networking Strategies for Improving Large Scientific Data Transfers\". <i>Proceedings of the 2018 IEEE International Conference on Communications Workshops<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FICCW.2018.8403576\" target=\"_blank\">10.1109\/ICCW.2018.8403576<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Named+Data+Networking+Strategies+for+Improving+Large+Scientific+Data+Transfers&rft.jtitle=Proceedings+of+the+2018+IEEE+International+Conference+on+Communications+Workshops&rft.aulast=Shannigrahi%2C+S.%3B+Fan%2C+C.%3B+Papadopolous%2C+C.+et+al.&rft.au=Shannigrahi%2C+S.%3B+Fan%2C+C.%3B+Papadopolous%2C+C.+et+al.&rft.date=2018&rft_id=info:doi\/10.1109%2FICCW.2018.8403576&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HadishSystems20-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HadishSystems20_19-0\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-HadishSystems20_19-1\">19.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Hadish, J.; Biggs, T.; Shealy, B. et al. (22 January 2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/zenodo.org\/record\/3620945\" target=\"_blank\">\"SystemsGenetics\/GEMmaker: Release v1.1\"<\/a>. <i>Zenodo<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5281%2Fzenodo.3620945\" target=\"_blank\">10.5281\/zenodo.3620945<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/zenodo.org\/record\/3620945\" target=\"_blank\">https:\/\/zenodo.org\/record\/3620945<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SystemsGenetics%2FGEMmaker%3A+Release+v1.1&rft.atitle=Zenodo&rft.aulast=Hadish%2C+J.%3B+Biggs%2C+T.%3B+Shealy%2C+B.+et+al.&rft.au=Hadish%2C+J.%3B+Biggs%2C+T.%3B+Shealy%2C+B.+et+al.&rft.date=22+January+2020&rft_id=info:doi\/10.5281%2Fzenodo.3620945&rft_id=https%3A%2F%2Fzenodo.org%2Frecord%2F3620945&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhangNamed14-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ZhangNamed14_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-ZhangNamed14_20-1\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhang, L.; Afanasyev, A.; Burke, J. et al. (2014). \"Named data networking\". <i>ACM SIGCOMM Computer Communication Review<\/i> <b>44<\/b> (3): 66-73. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2656877.2656887\" target=\"_blank\">10.1145\/2656877.2656887<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Named+data+networking&rft.jtitle=ACM+SIGCOMM+Computer+Communication+Review&rft.aulast=Zhang%2C+L.%3B+Afanasyev%2C+A.%3B+Burke%2C+J.+et+al.&rft.au=Zhang%2C+L.%3B+Afanasyev%2C+A.%3B+Burke%2C+J.+et+al.&rft.date=2014&rft.volume=44&rft.issue=3&rft.pages=66-73&rft_id=info:doi\/10.1145%2F2656877.2656887&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Hoque_NLSR13-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Hoque_NLSR13_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hoque, A.K.M.M.; Amin, S.O.; Alyyan, A. et al. (2013). \"NLSR: Named-data link state routing protocol\". <i>Proceedings of the 3rd ACM SIGCOMM workshop on Information-centric networking<\/i>: 15\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2491224.2491231\" target=\"_blank\">10.1145\/2491224.2491231<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=NLSR%3A+Named-data+link+state+routing+protocol&rft.jtitle=Proceedings+of+the+3rd+ACM+SIGCOMM+workshop+on+Information-centric+networking&rft.aulast=Hoque%2C+A.K.M.M.%3B+Amin%2C+S.O.%3B+Alyyan%2C+A.+et+al.&rft.au=Hoque%2C+A.K.M.M.%3B+Amin%2C+S.O.%3B+Alyyan%2C+A.+et+al.&rft.date=2013&rft.pages=15%E2%80%9320&rft_id=info:doi\/10.1145%2F2491224.2491231&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AfanasyevNFD16-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AfanasyevNFD16_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-AfanasyevNFD16_22-1\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Afanasyev, A.; Shi, J.; Zhang, B. et al. (October 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/named-data.net\/publications\/techreports\/ndn-0021-7-nfd-developer-guide\/\" target=\"_blank\">\"NFD Developer\u2019s Guide - Revision 7\"<\/a>. <i>NDN, Technical Report NDN-0021<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/named-data.net\/publications\/techreports\/ndn-0021-7-nfd-developer-guide\/\" target=\"_blank\">https:\/\/named-data.net\/publications\/techreports\/ndn-0021-7-nfd-developer-guide\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NFD+Developer%E2%80%99s+Guide+-+Revision+7&rft.atitle=NDN%2C+Technical+Report+NDN-0021&rft.aulast=Afanasyev%2C+A.%3B+Shi%2C+J.%3B+Zhang%2C+B.+et+al.&rft.au=Afanasyev%2C+A.%3B+Shi%2C+J.%3B+Zhang%2C+B.+et+al.&rft.date=October+2016&rft_id=https%3A%2F%2Fnamed-data.net%2Fpublications%2Ftechreports%2Fndn-0021-7-nfd-developer-guide%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SoNamed13-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SoNamed13_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">So, W.; Narayanan, A.; Oran, D. et al. (2013). \"Named data networking on a router: Forwarding at 20gbps and beyond\". <i>ACM SIGCOMM Computer Communication Review<\/i> <b>43<\/b> (4): 495\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2534169.2491699\" target=\"_blank\">10.1145\/2534169.2491699<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Named+data+networking+on+a+router%3A+Forwarding+at+20gbps+and+beyond&rft.jtitle=ACM+SIGCOMM+Computer+Communication+Review&rft.aulast=So%2C+W.%3B+Narayanan%2C+A.%3B+Oran%2C+D.+et+al.&rft.au=So%2C+W.%3B+Narayanan%2C+A.%3B+Oran%2C+D.+et+al.&rft.date=2013&rft.volume=43&rft.issue=4&rft.pages=495%E2%80%936&rft_id=info:doi\/10.1145%2F2534169.2491699&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhoussiPerform19-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KhoussiPerform19_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khoussi, S.; Nouri, A.; Shi, J. et al. (2019). \"Performance Evaluation of the NDN Data Plane Using Statistical Model Checking\". <i>Proceedings of the International Symposium on Automated Technology for Verification and Analysis<\/i>: 534\u201350. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-030-31784-3_31\" target=\"_blank\">10.1007\/978-3-030-31784-3_31<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Performance+Evaluation+of+the+NDN+Data+Plane+Using+Statistical+Model+Checking&rft.jtitle=Proceedings+of+the+International+Symposium+on+Automated+Technology+for+Verification+and+Analysis&rft.aulast=Khoussi%2C+S.%3B+Nouri%2C+A.%3B+Shi%2C+J.+et+al.&rft.au=Khoussi%2C+S.%3B+Nouri%2C+A.%3B+Shi%2C+J.+et+al.&rft.date=2019&rft.pages=534%E2%80%9350&rft_id=info:doi\/10.1007%2F978-3-030-31784-3_31&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChenNDN14-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChenNDN14_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Chen, S.; Shi, W; Cao, A. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/learn.tsinghua.edu.cn:8080\/2011011088\/WeiqiShi\/content\/NDNRepo.pdf\" target=\"_blank\">\"NDN Repo: An NDN Persistent Storage Model\"<\/a> (PDF). Tsinghua University<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/learn.tsinghua.edu.cn:8080\/2011011088\/WeiqiShi\/content\/NDNRepo.pdf\" target=\"_blank\">http:\/\/learn.tsinghua.edu.cn:8080\/2011011088\/WeiqiShi\/content\/NDNRepo.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NDN+Repo%3A+An+NDN+Persistent+Storage+Model&rft.atitle=&rft.aulast=Chen%2C+S.%3B+Shi%2C+W%3B+Cao%2C+A.+et+al.&rft.au=Chen%2C+S.%3B+Shi%2C+W%3B+Cao%2C+A.+et+al.&rft.date=2014&rft.pub=Tsinghua+University&rft_id=http%3A%2F%2Flearn.tsinghua.edu.cn%3A8080%2F2011011088%2FWeiqiShi%2Fcontent%2FNDNRepo.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DDBJHome-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DDBJHome_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ddbj.nig.ac.jp\/index-e.html\" target=\"_blank\">\"DDBJ\"<\/a>. Research Organization of Information and Systems. 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ddbj.nig.ac.jp\/index-e.html\" target=\"_blank\">https:\/\/www.ddbj.nig.ac.jp\/index-e.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 October 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=DDBJ&rft.atitle=&rft.date=2019&rft.pub=Research+Organization+of+Information+and+Systems&rft_id=https%3A%2F%2Fwww.ddbj.nig.ac.jp%2Findex-e.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EMBL-EBIHome-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EMBL-EBIHome_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ebi.ac.uk\/\" target=\"_blank\">\"EMBL-EBI\"<\/a>. Wellcome Genome Campus. 2021<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ebi.ac.uk\/\" target=\"_blank\">https:\/\/www.ebi.ac.uk\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=EMBL-EBI&rft.atitle=&rft.date=2021&rft.pub=Wellcome+Genome+Campus&rft_id=https%3A%2F%2Fwww.ebi.ac.uk%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LatheGenomic08-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LatheGenomic08_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lathe, W.; williams, J.; Mangan, M. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nature.com\/scitable\/topicpage\/genomic-data-resources-challenges-and-promises-743721\/\" target=\"_blank\">\"Genomic Data Resources: Challenges and Promises\"<\/a>. <i>Nature Education<\/i> <b>1<\/b> (3): 2<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nature.com\/scitable\/topicpage\/genomic-data-resources-challenges-and-promises-743721\/\" target=\"_blank\">https:\/\/www.nature.com\/scitable\/topicpage\/genomic-data-resources-challenges-and-promises-743721\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genomic+Data+Resources%3A+Challenges+and+Promises&rft.jtitle=Nature+Education&rft.aulast=Lathe%2C+W.%3B+williams%2C+J.%3B+Mangan%2C+M.+et+al.&rft.au=Lathe%2C+W.%3B+williams%2C+J.%3B+Mangan%2C+M.+et+al.&rft.date=2008&rft.volume=1&rft.issue=3&rft.pages=2&rft_id=https%3A%2F%2Fwww.nature.com%2Fscitable%2Ftopicpage%2Fgenomic-data-resources-challenges-and-promises-743721%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DankarTheDevel18-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DankarTheDevel18_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dankar, F.K.; Ptisyn, A.; Dankar, S.K. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5894154\" target=\"_blank\">\"The development of large-scale de-identified biomedical databases in the age of genomics\u2014principles and challenges\"<\/a>. <i>Human Genomics<\/i> <b>12<\/b>: 19. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs40246-018-0147-5\" target=\"_blank\">10.1186\/s40246-018-0147-5<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5894154\/\" target=\"_blank\">PMC5894154<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29636096\" target=\"_blank\">29636096<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5894154\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5894154<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+development+of+large-scale+de-identified+biomedical+databases+in+the+age+of+genomics%E2%80%94principles+and+challenges&rft.jtitle=Human+Genomics&rft.aulast=Dankar%2C+F.K.%3B+Ptisyn%2C+A.%3B+Dankar%2C+S.K.&rft.au=Dankar%2C+F.K.%3B+Ptisyn%2C+A.%3B+Dankar%2C+S.K.&rft.date=2018&rft.volume=12&rft.pages=19&rft_id=info:doi\/10.1186%2Fs40246-018-0147-5&rft_id=info:pmc\/PMC5894154&rft_id=info:pmid\/29636096&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5894154&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NationalData20-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NationalData20_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">National Genomics Data Center Members and Partners (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7145560\" target=\"_blank\">\"Database Resources of the National Genomics Data Center in 2020\"<\/a>. <i>Nucleic Acids Research<\/i> <b>48<\/b> (D1): D24\u201333. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkz913\" target=\"_blank\">10.1093\/nar\/gkz913<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7145560\/\" target=\"_blank\">PMC7145560<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31702008\" target=\"_blank\">31702008<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7145560\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7145560<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Database+Resources+of+the+National+Genomics+Data+Center+in+2020&rft.jtitle=Nucleic+Acids+Research&rft.aulast=National+Genomics+Data+Center+Members+and+Partners&rft.au=National+Genomics+Data+Center+Members+and+Partners&rft.date=2020&rft.volume=48&rft.issue=D1&rft.pages=D24%E2%80%9333&rft_id=info:doi\/10.1093%2Fnar%2Fgkz913&rft_id=info:pmc\/PMC7145560&rft_id=info:pmid\/31702008&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7145560&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LHCStudyGroupDesign91-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LHCStudyGroupDesign91_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">The LHC Study Group (May 1991). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/cds.cern.ch\/record\/220493\" target=\"_blank\">\"Design Study of the Large Hadron Collider (LHC): A Multiparticle Collider in the LEP Tunnel\"<\/a>. CERN<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/cds.cern.ch\/record\/220493\" target=\"_blank\">https:\/\/cds.cern.ch\/record\/220493<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Design+Study+of+the+Large+Hadron+Collider+%28LHC%29%3A+A+Multiparticle+Collider+in+the+LEP+Tunnel&rft.atitle=&rft.aulast=The+LHC+Study+Group&rft.au=The+LHC+Study+Group&rft.date=May+1991&rft.pub=CERN&rft_id=https%3A%2F%2Fcds.cern.ch%2Frecord%2F220493&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorCMIP5_10-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorCMIP5_10_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Taylor, K.E.; Doutriaux, C. (28 April 2010). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/pcmdi.llnl.gov\/mips\/cmip5\/docs\/CMIP5_output_metadata_requirements28Apr10.pdf?id=89\" target=\"_blank\">\"CMIP5 Model Output Requirements: File Contents and Format, Data Structure and Metadata\"<\/a> (PDF). PCMDI<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/pcmdi.llnl.gov\/mips\/cmip5\/docs\/CMIP5_output_metadata_requirements28Apr10.pdf?id=89\" target=\"_blank\">https:\/\/pcmdi.llnl.gov\/mips\/cmip5\/docs\/CMIP5_output_metadata_requirements28Apr10.pdf?id=89<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CMIP5+Model+Output+Requirements%3A+File+Contents+and+Format%2C+Data+Structure+and+Metadata&rft.atitle=&rft.aulast=Taylor%2C+K.E.%3B+Doutriaux%2C+C.&rft.au=Taylor%2C+K.E.%3B+Doutriaux%2C+C.&rft.date=28+April+2010&rft.pub=PCMDI&rft_id=https%3A%2F%2Fpcmdi.llnl.gov%2Fmips%2Fcmip5%2Fdocs%2FCMIP5_output_metadata_requirements28Apr10.pdf%3Fid%3D89&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MunsonBulk13-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MunsonBulk13_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Munson, M.C.; Simu, S. (11 November 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/scienceon.kisti.re.kr\/srch\/selectPORSrchPatent.do?cn=USP2015038996945\" target=\"_blank\">\"Bulk data transfer\"<\/a>. <i>ScienceON<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/scienceon.kisti.re.kr\/srch\/selectPORSrchPatent.do?cn=USP2015038996945\" target=\"_blank\">https:\/\/scienceon.kisti.re.kr\/srch\/selectPORSrchPatent.do?cn=USP2015038996945<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bulk+data+transfer&rft.atitle=ScienceON&rft.aulast=Munson%2C+M.C.%3B+Simu%2C+S.&rft.au=Munson%2C+M.C.%3B+Simu%2C+S.&rft.date=11+November+2013&rft_id=https%3A%2F%2Fscienceon.kisti.re.kr%2Fsrch%2FselectPORSrchPatent.do%3Fcn%3DUSP2015038996945&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IBMAspera-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IBMAspera_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ibm.com\/products\/aspera\" target=\"_blank\">\"Aspera\"<\/a>. IBM<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ibm.com\/products\/aspera\" target=\"_blank\">https:\/\/www.ibm.com\/products\/aspera<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Aspera&rft.atitle=&rft.pub=IBM&rft_id=https%3A%2F%2Fwww.ibm.com%2Fproducts%2Faspera&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UoCGlobus-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UoCGlobus_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.globus.org\/\" target=\"_blank\">\"Globus\"<\/a>. University of Chicago<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.globus.org\/\" target=\"_blank\">https:\/\/www.globus.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Globus&rft.atitle=&rft.pub=University+of+Chicago&rft_id=https%3A%2F%2Fwww.globus.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RajesekariRODS10-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RajesekariRODS10_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rajasekar, A.; Moore, R.; Hou, C.-Y. et al. (2010). \"iRODS Primer: Integrated Rule-Oriented Data System\". <i>Synthesis Lectures on Information Concepts, Retrieval, and Services<\/i> <b>2<\/b>: 1\u2013143. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2200%2FS00233ED1V01Y200912ICR012\" target=\"_blank\">10.2200\/S00233ED1V01Y200912ICR012<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=iRODS+Primer%3A+Integrated+Rule-Oriented+Data+System&rft.jtitle=Synthesis+Lectures+on+Information+Concepts%2C+Retrieval%2C+and+Services&rft.aulast=Rajasekar%2C+A.%3B+Moore%2C+R.%3B+Hou%2C+C.-Y.+et+al.&rft.au=Rajasekar%2C+A.%3B+Moore%2C+R.%3B+Hou%2C+C.-Y.+et+al.&rft.date=2010&rft.volume=2&rft.pages=1%E2%80%93143&rft_id=info:doi\/10.2200%2FS00233ED1V01Y200912ICR012&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChiangImplem11-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChiangImplem11_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chiang, G.-T.; Clapham, P.; Qi, G. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3228552\" target=\"_blank\">\"Implementing a genomic data management system using iRODS in the Wellcome Trust Sanger Institute\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b>: 361. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-361\" target=\"_blank\">10.1186\/1471-2105-12-361<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3228552\/\" target=\"_blank\">PMC3228552<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21906284\" target=\"_blank\">21906284<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3228552\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3228552<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementing+a+genomic+data+management+system+using+iRODS+in+the+Wellcome+Trust+Sanger+Institute&rft.jtitle=BMC+Bioinformatics&rft.aulast=Chiang%2C+G.-T.%3B+Clapham%2C+P.%3B+Qi%2C+G.+et+al.&rft.au=Chiang%2C+G.-T.%3B+Clapham%2C+P.%3B+Qi%2C+G.+et+al.&rft.date=2011&rft.volume=12&rft.pages=361&rft_id=info:doi\/10.1186%2F1471-2105-12-361&rft_id=info:pmc\/PMC3228552&rft_id=info:pmid\/21906284&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3228552&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EilbeckTheSeq05-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EilbeckTheSeq05_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Eilbeck, K.; Lewis, S.E.; Mungall, C.J. et al. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1175956\" target=\"_blank\">\"The Sequence Ontology: A tool for the unification of genome annotations\"<\/a>. <i>Genome Biology<\/i> <b>6<\/b> (5): R44. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2005-6-5-r44\" target=\"_blank\">10.1186\/gb-2005-6-5-r44<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1175956\/\" target=\"_blank\">PMC1175956<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15892872\" target=\"_blank\">15892872<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1175956\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1175956<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Sequence+Ontology%3A+A+tool+for+the+unification+of+genome+annotations&rft.jtitle=Genome+Biology&rft.aulast=Eilbeck%2C+K.%3B+Lewis%2C+S.E.%3B+Mungall%2C+C.J.+et+al.&rft.au=Eilbeck%2C+K.%3B+Lewis%2C+S.E.%3B+Mungall%2C+C.J.+et+al.&rft.date=2005&rft.volume=6&rft.issue=5&rft.pages=R44&rft_id=info:doi\/10.1186%2Fgb-2005-6-5-r44&rft_id=info:pmc\/PMC1175956&rft_id=info:pmid\/15892872&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1175956&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchrimlDisease12-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchrimlDisease12_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schriml, L.M.; Arze, C.; Nadendla, S. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3245088\" target=\"_blank\">\"Disease Ontology: A backbone for disease semantic integration\"<\/a>. <i>Nucleic Acids Research<\/i> <b>40<\/b> (DB 1): D940-6. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkr972\" target=\"_blank\">10.1093\/nar\/gkr972<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3245088\/\" target=\"_blank\">PMC3245088<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22080554\" target=\"_blank\">22080554<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3245088\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3245088<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Disease+Ontology%3A+A+backbone+for+disease+semantic+integration&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Schriml%2C+L.M.%3B+Arze%2C+C.%3B+Nadendla%2C+S.+et+al.&rft.au=Schriml%2C+L.M.%3B+Arze%2C+C.%3B+Nadendla%2C+S.+et+al.&rft.date=2012&rft.volume=40&rft.issue=DB+1&rft.pages=D940-6&rft_id=info:doi\/10.1093%2Fnar%2Fgkr972&rft_id=info:pmc\/PMC3245088&rft_id=info:pmid\/22080554&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3245088&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GOCGene15-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GOCGene15_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gene Ontology Consortium (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383973\" target=\"_blank\">\"Gene Ontology Consortium: Going forward\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (DB 1): D1049\u201356. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1179\" target=\"_blank\">10.1093\/nar\/gku1179<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383973\/\" target=\"_blank\">PMC4383973<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25428369\" target=\"_blank\">25428369<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383973\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383973<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+Ontology+Consortium%3A+Going+forward&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Gene+Ontology+Consortium&rft.au=Gene+Ontology+Consortium&rft.date=2015&rft.volume=43&rft.issue=DB+1&rft.pages=D1049%E2%80%9356&rft_id=info:doi\/10.1093%2Fnar%2Fgku1179&rft_id=info:pmc\/PMC4383973&rft_id=info:pmid\/25428369&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383973&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShannigrahiWhat20-41\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShannigrahiWhat20_41-0\">41.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShannigrahiWhat20_41-1\">41.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shannigrahi, S.; Fan, C.; Partridge, C. (2020). \"What's in a Name?: Naming Big Science Data in Named Data Networking\". <i>Proceedings of the 7th ACM Conference on Information-Centric Networking<\/i>: 12\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3405656.3418717\" target=\"_blank\">10.1145\/3405656.3418717<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What%27s+in+a+Name%3F%3A+Naming+Big+Science+Data+in+Named+Data+Networking&rft.jtitle=Proceedings+of+the+7th+ACM+Conference+on+Information-Centric+Networking&rft.aulast=Shannigrahi%2C+S.%3B+Fan%2C+C.%3B+Partridge%2C+C.&rft.au=Shannigrahi%2C+S.%3B+Fan%2C+C.%3B+Partridge%2C+C.&rft.date=2020&rft.pages=12%E2%80%9323&rft_id=info:doi\/10.1145%2F3405656.3418717&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShannigrahiTheFut19-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShannigrahiTheFut19_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Shannigrahi, S. (Summer 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/mountainscholar.org\/bitstream\/handle\/10217\/197325\/Shannigrahi_colostate_0053A_15549.pdf?sequence=1&isAllowed=y\" target=\"_blank\">\"The Future of Networking Is the Future of Big Data\"<\/a>. Colorado State University<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/mountainscholar.org\/bitstream\/handle\/10217\/197325\/Shannigrahi_colostate_0053A_15549.pdf?sequence=1&isAllowed=y\" target=\"_blank\">https:\/\/mountainscholar.org\/bitstream\/handle\/10217\/197325\/Shannigrahi_colostate_0053A_15549.pdf?sequence=1&isAllowed=y<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Future+of+Networking+Is+the+Future+of+Big+Data&rft.atitle=&rft.aulast=Shannigrahi%2C+S.&rft.au=Shannigrahi%2C+S.&rft.date=Summer+2019&rft.pub=Colorado+State+University&rft_id=https%3A%2F%2Fmountainscholar.org%2Fbitstream%2Fhandle%2F10217%2F197325%2FShannigrahi_colostate_0053A_15549.pdf%3Fsequence%3D1%26isAllowed%3Dy&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TierneyHigh10-43\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TierneyHigh10_43-0\">43.0<\/a><\/sup> <sup><a href=\"#cite_ref-TierneyHigh10_43-1\">43.1<\/a><\/sup> <sup><a href=\"#cite_ref-TierneyHigh10_43-2\">43.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Tierney, B.; Metzger, J. (July 2010). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/fasterdata.es.net\/assets\/fasterdata\/JT-201010.pdf\" target=\"_blank\">\"Higher Performance Bulk Data Transfer\"<\/a>. ESnet<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/fasterdata.es.net\/assets\/fasterdata\/JT-201010.pdf\" target=\"_blank\">https:\/\/fasterdata.es.net\/assets\/fasterdata\/JT-201010.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Higher+Performance+Bulk+Data+Transfer&rft.atitle=&rft.aulast=Tierney%2C+B.%3B+Metzger%2C+J.&rft.au=Tierney%2C+B.%3B+Metzger%2C+J.&rft.date=July+2010&rft.pub=ESnet&rft_id=https%3A%2F%2Ffasterdata.es.net%2Fassets%2Ffasterdata%2FJT-201010.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MillsMoving19-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MillsMoving19_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mills, N.; Bensman, E.M.; Poehlman, W.L. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6572328\" target=\"_blank\">\"Moving Just Enough Deep Sequencing Data to Get the Job Done\"<\/a>. <i>Bioinformatics and Biology Insights<\/i> <b>13<\/b>: 1177932219856359. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1177932219856359\" target=\"_blank\">10.1177\/1177932219856359<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6572328\/\" target=\"_blank\">PMC6572328<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31236009\" target=\"_blank\">31236009<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6572328\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6572328<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Moving+Just+Enough+Deep+Sequencing+Data+to+Get+the+Job+Done&rft.jtitle=Bioinformatics+and+Biology+Insights&rft.aulast=Mills%2C+N.%3B+Bensman%2C+E.M.%3B+Poehlman%2C+W.L.+et+al.&rft.au=Mills%2C+N.%3B+Bensman%2C+E.M.%3B+Poehlman%2C+W.L.+et+al.&rft.date=2019&rft.volume=13&rft.pages=1177932219856359&rft_id=info:doi\/10.1177%2F1177932219856359&rft_id=info:pmc\/PMC6572328&rft_id=info:pmid\/31236009&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6572328&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YuSchema15-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YuSchema15_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yu, Y.; Afanasyev, A.; Clark, D. et al. (2015). \"Schematizing Trust in Named Data Networking\". <i>Proceedings of the 2nd ACM Conference on Information-Centric Networking<\/i>: 177\u2013186. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2810156.2810170\" target=\"_blank\">10.1145\/2810156.2810170<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Schematizing+Trust+in+Named+Data+Networking&rft.jtitle=Proceedings+of+the+2nd+ACM+Conference+on+Information-Centric+Networking&rft.aulast=Yu%2C+Y.%3B+Afanasyev%2C+A.%3B+Clark%2C+D.+et+al.&rft.au=Yu%2C+Y.%3B+Afanasyev%2C+A.%3B+Clark%2C+D.+et+al.&rft.date=2015&rft.pages=177%E2%80%93186&rft_id=info:doi\/10.1145%2F2810156.2810170&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ESnet-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ESnet_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/fasterdata.es.net\/\" target=\"_blank\">\"ESnet\"<\/a>. ESnet<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/fasterdata.es.net\/\" target=\"_blank\">https:\/\/fasterdata.es.net\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ESnet&rft.atitle=&rft.pub=ESnet&rft_id=https%3A%2F%2Ffasterdata.es.net%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ICANNWelcome-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ICANNWelcome_47-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.icann.org\/resources\/pages\/welcome-2012-02-25-en\" target=\"_blank\">\"Welcome to ICANN!\"<\/a>. ICANN<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.icann.org\/resources\/pages\/welcome-2012-02-25-en\" target=\"_blank\">https:\/\/www.icann.org\/resources\/pages\/welcome-2012-02-25-en<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 14 November 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Welcome+to+ICANN%21&rft.atitle=&rft.pub=ICANN&rft_id=https%3A%2F%2Fwww.icann.org%2Fresources%2Fpages%2Fwelcome-2012-02-25-en&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NSFSciDAS19-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NSFSciDAS19_48-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nsf.gov\/awardsearch\/showAward?AWD_ID=1659300\" target=\"_blank\">\"CC*Data: National Cyberinfrastructure for Scientific Data Analysis at Scale (SciDAS)\"<\/a>. National Science Foundation. 24 October 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nsf.gov\/awardsearch\/showAward?AWD_ID=1659300\" target=\"_blank\">https:\/\/www.nsf.gov\/awardsearch\/showAward?AWD_ID=1659300<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CC%2AData%3A+National+Cyberinfrastructure+for+Scientific+Data+Analysis+at+Scale+%28SciDAS%29&rft.atitle=&rft.date=24+October+2019&rft.pub=National+Science+Foundation&rft_id=https%3A%2F%2Fwww.nsf.gov%2Fawardsearch%2FshowAward%3FAWD_ID%3D1659300&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZerbinoEnsembl18-49\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZerbinoEnsembl18_49-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zerbino, D.R.; Achuthan, P.; Akanni, W. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5753206\" target=\"_blank\">\"Ensembl 2018\"<\/a>. <i>Nucleic Acids Research<\/i> <b>46<\/b> (D1): D754\u201361. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkx1098\" target=\"_blank\">10.1093\/nar\/gkx1098<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5753206\/\" target=\"_blank\">PMC5753206<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29155950\" target=\"_blank\">29155950<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5753206\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5753206<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ensembl+2018&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Zerbino%2C+D.R.%3B+Achuthan%2C+P.%3B+Akanni%2C+W.+et+al.&rft.au=Zerbino%2C+D.R.%3B+Achuthan%2C+P.%3B+Akanni%2C+W.+et+al.&rft.date=2018&rft.volume=46&rft.issue=D1&rft.pages=D754%E2%80%9361&rft_id=info:doi\/10.1093%2Fnar%2Fgkx1098&rft_id=info:pmc\/PMC5753206&rft_id=info:pmid\/29155950&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5753206&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Pynome-50\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Pynome_50-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/SystemsGenetics\/pynome\" target=\"_blank\">\"SystemsGenetics\/pynome\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/SystemsGenetics\/pynome\" target=\"_blank\">https:\/\/github.com\/SystemsGenetics\/pynome<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 February 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SystemsGenetics%2Fpynome&rft.atitle=GitHub&rft_id=https%3A%2F%2Fgithub.com%2FSystemsGenetics%2Fpynome&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OlschanowskySupporting14-51\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OlschanowskySupporting14_51-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Olschanowsky, C.; Shannigrahi, S.; Papadopoulos, C. (2014). \"Supporting climate research using named data networking\". <i>Proceedings of the 2014 IEEE 20th International Workshop on Local & Metropolitan Area Networks<\/i>: 1\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FLANMAN.2014.7028640\" target=\"_blank\">10.1109\/LANMAN.2014.7028640<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Supporting+climate+research+using+named+data+networking&rft.jtitle=Proceedings+of+the+2014+IEEE+20th+International+Workshop+on+Local+%26+Metropolitan+Area+Networks&rft.aulast=Olschanowsky%2C+C.%3B+Shannigrahi%2C+S.%3B+Papadopoulos%2C+C.&rft.au=Olschanowsky%2C+C.%3B+Shannigrahi%2C+S.%3B+Papadopoulos%2C+C.&rft.date=2014&rft.pages=1%E2%80%936&rft_id=info:doi\/10.1109%2FLANMAN.2014.7028640&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FicklinDiscov17-52\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FicklinDiscov17_52-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ficklin, S.P.; Dunwoodie, L.J.; Poehlman, W.L. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5561081\" target=\"_blank\">\"Discovering Condition-Specific Gene Co-Expression Patterns Using Gaussian Mixture Models: A Cancer Case Study\"<\/a>. <i>Scientific Reports<\/i> <b>7<\/b> (1): 8617. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41598-017-09094-4\" target=\"_blank\">10.1038\/s41598-017-09094-4<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5561081\/\" target=\"_blank\">PMC5561081<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28819158\" target=\"_blank\">28819158<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5561081\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5561081<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Discovering+Condition-Specific+Gene+Co-Expression+Patterns+Using+Gaussian+Mixture+Models%3A+A+Cancer+Case+Study&rft.jtitle=Scientific+Reports&rft.aulast=Ficklin%2C+S.P.%3B+Dunwoodie%2C+L.J.%3B+Poehlman%2C+W.L.+et+al.&rft.au=Ficklin%2C+S.P.%3B+Dunwoodie%2C+L.J.%3B+Poehlman%2C+W.L.+et+al.&rft.date=2017&rft.volume=7&rft.issue=1&rft.pages=8617&rft_id=info:doi\/10.1038%2Fs41598-017-09094-4&rft_id=info:pmc\/PMC5561081&rft_id=info:pmid\/28819158&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5561081&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RocheCancer17-53\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RocheCancer17_53-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Roche, K.; Feltus, F.A.; Park, J.P. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5464651\" target=\"_blank\">\"Cancer cell redirection biomarker discovery using a mutual information approach\"<\/a>. <i>PLoS One<\/i> <b>12<\/b> (6): e0179265. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0179265\" target=\"_blank\">10.1371\/journal.pone.0179265<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5464651\/\" target=\"_blank\">PMC5464651<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28594912\" target=\"_blank\">28594912<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5464651\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5464651<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cancer+cell+redirection+biomarker+discovery+using+a+mutual+information+approach&rft.jtitle=PLoS+One&rft.aulast=Roche%2C+K.%3B+Feltus%2C+F.A.%3B+Park%2C+J.P.+et+al.&rft.au=Roche%2C+K.%3B+Feltus%2C+F.A.%3B+Park%2C+J.P.+et+al.&rft.date=2017&rft.volume=12&rft.issue=6&rft.pages=e0179265&rft_id=info:doi\/10.1371%2Fjournal.pone.0179265&rft_id=info:pmc\/PMC5464651&rft_id=info:pmid\/28594912&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5464651&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DunwoodieDiscov18-54\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DunwoodieDiscov18_54-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dunwoodie, L.J.; Poehlman, W.L.; Ficklin, S.P. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5834250\" target=\"_blank\">\"Discovery and validation of a glioblastoma co-expressed gene module\"<\/a>. <i>Oncotarget<\/i> <b>9<\/b> (13): 10995-11008. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.18632%2Foncotarget.24228\" target=\"_blank\">10.18632\/oncotarget.24228<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5834250\/\" target=\"_blank\">PMC5834250<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29541392\" target=\"_blank\">29541392<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5834250\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5834250<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Discovery+and+validation+of+a+glioblastoma+co-expressed+gene+module&rft.jtitle=Oncotarget&rft.aulast=Dunwoodie%2C+L.J.%3B+Poehlman%2C+W.L.%3B+Ficklin%2C+S.P.+et+al.&rft.au=Dunwoodie%2C+L.J.%3B+Poehlman%2C+W.L.%3B+Ficklin%2C+S.P.+et+al.&rft.date=2018&rft.volume=9&rft.issue=13&rft.pages=10995-11008&rft_id=info:doi\/10.18632%2Foncotarget.24228&rft_id=info:pmc\/PMC5834250&rft_id=info:pmid\/29541392&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5834250&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PoehlmanLink19-55\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PoehlmanLink19_55-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Poehlman, W.L.; Hsieh, J.J.; Feltus, F.A. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6393532\" target=\"_blank\">\"Linking Binary Gene Relationships to Drivers of Renal Cell Carcinoma Reveals Convergent Function in Alternate Tumor Progression Paths\"<\/a>. <i>Scientific Reports<\/i> <b>9<\/b> (1): 2899. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41598-019-39875-y\" target=\"_blank\">10.1038\/s41598-019-39875-y<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6393532\/\" target=\"_blank\">PMC6393532<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30814637\" target=\"_blank\">30814637<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6393532\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6393532<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Linking+Binary+Gene+Relationships+to+Drivers+of+Renal+Cell+Carcinoma+Reveals+Convergent+Function+in+Alternate+Tumor+Progression+Paths&rft.jtitle=Scientific+Reports&rft.aulast=Poehlman%2C+W.L.%3B+Hsieh%2C+J.J.%3B+Feltus%2C+F.A.+et+al.&rft.au=Poehlman%2C+W.L.%3B+Hsieh%2C+J.J.%3B+Feltus%2C+F.A.+et+al.&rft.date=2019&rft.volume=9&rft.issue=1&rft.pages=2899&rft_id=info:doi\/10.1038%2Fs41598-019-39875-y&rft_id=info:pmc\/PMC6393532&rft_id=info:pmid\/30814637&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6393532&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NextFlow-56\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NextFlow_56-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/nextflow.io\/\" target=\"_blank\">\"Nextflow\"<\/a>. Seqera Labs<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/nextflow.io\/\" target=\"_blank\">https:\/\/nextflow.io\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 October 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Nextflow&rft.atitle=&rft.pub=Seqera+Labs&rft_id=https%3A%2F%2Fnextflow.io%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DiTommasoNext17-57\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DiTommasoNext17_57-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Di Tommaso, P.D.; Chatzou, M.; Floden, E.W. et al. (2017). \"Nextflow enables reproducible computational workflows\". <i>Nature Biotechnology<\/i> <b>35<\/b> (4): 316\u201319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt.3820\" target=\"_blank\">10.1038\/nbt.3820<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28398311\" target=\"_blank\">28398311<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Nextflow+enables+reproducible+computational+workflows&rft.jtitle=Nature+Biotechnology&rft.aulast=Di+Tommaso%2C+P.D.%3B+Chatzou%2C+M.%3B+Floden%2C+E.W.+et+al.&rft.au=Di+Tommaso%2C+P.D.%3B+Chatzou%2C+M.%3B+Floden%2C+E.W.+et+al.&rft.date=2017&rft.volume=35&rft.issue=4&rft.pages=316%E2%80%9319&rft_id=info:doi\/10.1038%2Fnbt.3820&rft_id=info:pmid\/28398311&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangOptimal13-58\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangOptimal13_58-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, Y.; Li, Z.; Tyson, G. et al. (2013). \"Optimal cache allocation for Content-Centric Networking\". <i>Proceedings of the 21st IEEE International Conference on Network Protocols<\/i>: 1\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FICNP.2013.6733577\" target=\"_blank\">10.1109\/ICNP.2013.6733577<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Optimal+cache+allocation+for+Content-Centric+Networking&rft.jtitle=Proceedings+of+the+21st+IEEE+International+Conference+on+Network+Protocols&rft.aulast=Wang%2C+Y.%3B+Li%2C+Z.%3B+Tyson%2C+G.+et+al.&rft.au=Wang%2C+Y.%3B+Li%2C+Z.%3B+Tyson%2C+G.+et+al.&rft.date=2013&rft.pages=1%E2%80%9310&rft_id=info:doi\/10.1109%2FICNP.2013.6733577&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FABRICHome-59\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FABRICHome_59-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/fabric-testbed.net\/\" target=\"_blank\">\"FABRIC\"<\/a>. RENCI<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/fabric-testbed.net\/\" target=\"_blank\">https:\/\/fabric-testbed.net\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 14 November 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=FABRIC&rft.atitle=&rft.pub=RENCI&rft_id=https%3A%2F%2Ffabric-testbed.net%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CockTheSanger10-60\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CockTheSanger10_60-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cock, P.J.A.; Fields, C.J.; Goto, N. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2847217\" target=\"_blank\">\"The Sanger FASTQ file format for sequences with quality scores, and the Solexa\/Illumina FASTQ variants\"<\/a>. <i>Nucleic Acids Research<\/i> <b>38<\/b> (6): 1767-71. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkp1137\" target=\"_blank\">10.1093\/nar\/gkp1137<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2847217\/\" target=\"_blank\">PMC2847217<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20015970\" target=\"_blank\">20015970<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2847217\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2847217<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Sanger+FASTQ+file+format+for+sequences+with+quality+scores%2C+and+the+Solexa%2FIllumina+FASTQ+variants&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Cock%2C+P.J.A.%3B+Fields%2C+C.J.%3B+Goto%2C+N.+et+al.&rft.au=Cock%2C+P.J.A.%3B+Fields%2C+C.J.%3B+Goto%2C+N.+et+al.&rft.date=2010&rft.volume=38&rft.issue=6&rft.pages=1767-71&rft_id=info:doi\/10.1093%2Fnar%2Fgkp1137&rft_id=info:pmc\/PMC2847217&rft_id=info:pmid\/20015970&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2847217&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmarrThePac18-61\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmarrThePac18_61-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smarr, L.; Crittenden, C.; DeFanti, T. et al. (2018). \"The Pacific Research Platform: Making High-Speed Networking a Reality for the Scientist\". <i>Proceedings of the Practice and Experience on Advanced Research Computing<\/i>: 1\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3219104.3219108\" target=\"_blank\">10.1145\/3219104.3219108<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Pacific+Research+Platform%3A+Making+High-Speed+Networking+a+Reality+for+the+Scientist&rft.jtitle=Proceedings+of+the+Practice+and+Experience+on+Advanced+Research+Computing&rft.aulast=Smarr%2C+L.%3B+Crittenden%2C+C.%3B+DeFanti%2C+T.+et+al.&rft.au=Smarr%2C+L.%3B+Crittenden%2C+C.%3B+DeFanti%2C+T.+et+al.&rft.date=2018&rft.pages=1%E2%80%938&rft_id=info:doi\/10.1145%2F3219104.3219108&rfr_id=info:sid\/en.wikipedia.org:Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original paper listed references alphabetically; this wiki lists them by order of appearance, by design. The two footnotes were turned into inline references for convenience.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194107\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.106 seconds\nReal time usage: 2.969 seconds\nPreprocessor visited node count: 44692\/1000000\nPreprocessor generated node count: 39930\/1000000\nPost\u2010expand include size: 339686\/2097152 bytes\nTemplate argument size: 115157\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 892.975 1 - -total\n 89.66% 800.629 1 - Template:Reflist\n 77.01% 687.666 61 - Template:Citation\/core\n 58.77% 524.787 41 - Template:Cite_journal\n 21.54% 192.365 20 - Template:Cite_web\n 7.60% 67.860 76 - Template:Citation\/identifier\n 4.32% 38.573 1 - Template:Infobox_journal_article\n 4.29% 38.334 65 - Template:Citation\/make_link\n 4.10% 36.632 1 - Template:Infobox\n 2.96% 26.462 170 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12436-0!*!0!!en!5!* and timestamp 20210429194104 and revision id 42029\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows\">https:\/\/www.limswiki.org\/index.php\/Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","784b63abe742a2cc6286e50d625220ec_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/2e\/Fig1_Ogle_FrontBigData2021_4.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b3\/Fig2_Ogle_FrontBigData2021_4.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/84\/Fig3_Ogle_FrontBigData2021_4.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/54\/Fig4_Ogle_FrontBigData2021_4.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ec\/Fig5_Ogle_FrontBigData2021_4.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d5\/Tab1_Ogle_FrontBigData2021_4.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/5a\/Fig6_Ogle_FrontBigData2021_4.jpg"],"784b63abe742a2cc6286e50d625220ec_timestamp":1619725264,"7d7600ccadb42ccd7bdbe90737c9955d_type":"article","7d7600ccadb42ccd7bdbe90737c9955d_title":"Kadi4Mat: A research data infrastructure for materials science (Brandt et al. 2021)","7d7600ccadb42ccd7bdbe90737c9955d_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science","7d7600ccadb42ccd7bdbe90737c9955d_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Kadi4Mat: A research data infrastructure for materials science\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nKadi4Mat: A research data infrastructure for materials scienceJournal\n \nData Science JournalAuthor(s)\n \nBrandt, Nico; Griem, Lars; Herrmann, Christoph; Schoof, Ephraim; Tosato, Giovanna; Zhao, Yinghan;\r\nZschumme, Philipp; Selzer, MichaelAuthor affiliation(s)\n \nKarlsruhe Institute of Technology, Karlsruhe University of Applied Sciences, Helmholtz Institute UlmPrimary contact\n \nEmail: nico dot brandt at kit dot eduYear published\n \n2021Volume and issue\n \n20(1)Article #\n \n8DOI\n \n10.5334\/dsj-2021-008ISSN\n \n1683-1470Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/datascience.codata.org\/articles\/10.5334\/dsj-2021-008\/Download\n \nhttps:\/\/datascience.codata.org\/articles\/10.5334\/dsj-2021-008\/galley\/1048\/download\/ (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Concepts \n\n3.1 Electronic laboratory notebook \n3.2 Repository \n\n\n4 Implementation \n5 Results \n6 Conclusion \n7 Acknowledgements \n\n7.1 Competing interests \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nThe concepts and current developments of a research data infrastructure for materials science are presented, extending and combining the features of an electronic laboratory notebook (ELN) and a repository. The objective of this infrastructure is to incorporate the possibility of structured data storage and data exchange with documented and reproducible data analysis and visualization, which finally leads to the publication of the data. This way, researchers can be supported throughout the entire research process. The software is being developed as a web-based and desktop-based system, offering both a graphical user interface (GUI) and a programmatic interface. The focus of the development is on the integration of technologies and systems based on both established as well as new concepts. Due to the heterogeneous nature of materials science data, the current features are kept mostly generic, and the structuring of the data is largely left to the users. As a result, an extension of the research data infrastructure to other disciplines is possible in the future. The source code of the project is publicly available under a permissive Apache 2.0 license.\nKeywords: research data management, electronic laboratory notebook, repository, open source, materials science\n\nIntroduction \nIn the engineering sciences, the handling of digital research data plays an increasingly important role in all fields of application.[1] This is especially the case, due to the growing amount of data obtained from experiments and simulations.[2] The extraction of knowledge from these data is referred to as a data-driven, fourth paradigm of science, filed under the keyword \"data science.\"[3] This is particularly true in materials science, as the research and understanding of new materials are becoming more and more complex.[4] Without suitable analysis methods, the ever-growing amount of data will no longer be manageable. In order to be able to perform appropriate data analyses smoothly, the structured storage of research data and associated metadata is an important aspect. Specifically, a uniform research data management is needed, which is made possible by appropriate infrastructures such as research data repositories. In addition to uniform data storage, such systems can help to overcome inter-institutional hurdles in data exchange, compare theoretical and experimental data, and provide reproducible workflows for data analysis. Furthermore, linking the data with persistent identifiers enables other researchers to directly reference them in their work.\nIn particular, repositories for the storage and internal or public exchange of research data are becoming increasingly widespread. In particular, the publication of such data, either on its own or as a supplement to a text publication, is increasingly encouraged or sometimes even required.[5] In order to find a suitable repository, services such as re3data.org[6] or FAIRSharing[7] are available. These services also make it possible to find subject-specific repositories for materials science data. Two well-known examples are the Materials Project[8] and the NOMAD Repository.[9] Indexed repositories are usually hosted centrally or institutionally and are mostly used for the publication of data. However, some of the underlying systems can also be installed by the user, e.g., for internal use within individual research groups. Additionally, this allows full control over stored data as well as internal data exchanges, if this function is not already part of the repository. In this respect, open-source systems are particularly important, as this means independence from vendors and opens up the possibility of modifying the existing functionality or adding additional features, sometimes via built-in plug-in systems. Examples of such systems are CKAN[10], Dataverse[11], DSpace[12], and Invenio[13], where the latter is the basis of Zenodo.[14] The listed repositories are all generic and represent only a selection of the existing open-source systems.[15]\nIn addition to repositories, a second type of system increasingly being used in experiment-oriented research areas is the electronic laboratory notebook (ELN).[16] Nowadays, the functionality of ELNs goes far beyond the simple replacement of paper-based laboratory notebooks, and can also include aspects such as data analysis, as seen, for example, in Galaxy[17] or Jupyter Notebook.[18] Both systems focus primarily on providing accessible and reproducible computational research data. However, the boundary between unstructured and structured data is increasingly becoming blurred, the latter being traditionally only found in laboratory information management systems (LIMS).[19][20][21] Most existing ELNs are domain-specific and limited to research disciplines such as biology or chemistry.[21] According to current knowledge, a system specifically tailored to materials science does not exist. For ELNs, there are also open-source systems such as eLabFTW[22], sciNote[23], or Chemotion.[24] Compared to the repositories, however, the selection of ELNs is smaller. Furthermore, only the first two mentioned systems are generic.\nThus, generic research data systems and software are available for both ELNs and repositories, which, in principle, could also be used in materials science. The listed open-source solutions are of particular relevance, as they can be adapted to different needs and are generally suitable for use in a custom installation within single research groups. However, both aspects can be a considerable hurdle, especially for smaller groups. Due to a lack of resources, structured research data management and the possibility of making data available for subsequent use is therefore particularly difficult for such groups.[25] What is finally missing is a system that can be deployed and used both centrally and decentrally, as well as internally and publicly, without major obstacles. The system should support researchers throughout the entire research process, starting with the generation and extraction of raw data, up to the structured storage, exchange, and analysis of the data, resulting in the final publication of the corresponding results. In this way, the features of the ELN and the repository are combined, creating a virtual research environment[26] that accelerates the generation of innovations by facilitating collaboration among researchers. In an interdisciplinary field like materials science, there is a special need to model the very heterogeneous workflows of researchers.[4]\nFor this purpose, the research data infrastructure Kadi4Mat (Karlsruhe Data Infrastructure for Materials Sciences) is being developed at the Institute for Applied Materials (IAM-CMS) of the Karlsruhe Institute of Technology (KIT). The aim of the software is to combine the possibility of structured data storage with documented and reproducible workflows for data analysis and visualization tasks, incorporating new concepts with established technologies and existing solutions. In the development of the software, the FAIR Guiding Principles[27] for scientific data management are taken into account. Instances of the data infrastructure have already been deployed and today show how structured data storage and data exchange are made possible.[28] Furthermore, the source code of the project is publicly available under a permissive Apache 2.0 license.[29]\n\nConcepts \nKadi4Mat is logically divided into the two components\u2014an ELN and a repository\u2014which have access to various tools and technical infrastructures. The components can be used by web- and desktop-based applications, via uniform interfaces. Both a graphical and a programmatic interface are provided, using machine-readable formats and various exchange protocols. In Figure 1, a conceptual overview of the infrastructure of Kadi4Mat is presented.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Conceptual overview of the infrastructure of Kadi4Mat. The system is logically divided into the two components\u2014an ELN and a repository\u2014which have access to various data handling tools and technical infrastructures. The two components can be used both graphically and programmatically via uniform interfaces.\n\n\n\nElectronic laboratory notebook \nThe so-called workflows are of particular importance in the ELN component. A \"workflow\" is a generic concept that describes a well-defined sequence of sequential or parallel steps, which are processed as automatically as possible. This can include the execution of an analysis tool or the control and data retrieval of an experimental device. To accommodate such heterogeneity, concrete steps must be implemented as flexibly as possible, since they are highly user- and application-specific. In Figure 1, the types of tools shown in the second layer are used as part of the workflows, so as to implement the actual functionality of the various steps. These can be roughly divided into analysis, visualization, transformation, and transportation tasks. In order to keep the application of these tools as generic as possible, a combination of provided and user-defined tools is accessed. From a user\u2019s perspective, it must be possible to provide such tools in an easy manner, while the execution of each tool must take place in a secure and functional environment. This is especially true for existing tools\u2014e.g., a simple MATLAB[30] script\u2014which require certain dependencies to be executed and must be equipped with a suitable interface to be used within a workflow. Depending on their functionality, the tools must in turn access various technical infrastructures. In addition to the use of the repository and computing infrastructure, direct access to devices is also important for more complex data analyses. The automation of a typical workflow of experimenters is only fully possible if data and metadata, created by devices, can be captured. However, such an integration is not trivial due to a heterogeneous device landscape, as well as proprietary data formats and interfaces.[31][32] In Kadi4Mat, it should also be possible to use individual tools separately, where appropriate, i.e., outside a workflow. For example, when using the web-based interface, a visualization tool for a custom data format may be used to generate a preview of a datum that can be directly displayed in a web browser.\nIn Figure 2, the current concept for the integration of workflows in Kadi4Mat is shown. Different steps of a workflow can be defined with a graphical node editor. Either a web-based or a desktop-based version of such an editor can be used, the latter running as an ordinary application on a local workstation. With the help of such an editor, the different steps or tools to be executed are defined, linked, and, most importantly, parameterized. The execution of a workflow can be started via an external component called \"Process Manager.\" This component in turn manages several process engines, which take care of executing the workflows. The process engines potentially differ in their implementation and functionality. A simple process engine, for example, could be limited to a sequential execution order of the different tasks, while another one could execute independent tasks in parallel. All engines process the required steps based on the information stored in the workflow. With appropriate transport tools, the data and metadata required for each step, as well as the resulting output, can be exported or imported from Kadi4Mat using the existing interfaces of the research data infrastructure. With similar tools, the use of other external data sources becomes possible, and with it the possibility to handle large amounts of data via suitable exchange protocols. The use of locally stored data is also possible when running a workflow on a local workstation.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Conceptual overview of the workflow architecture. Each workflow is defined using a graphical editor that is either directly integrated into the web-based interface of Kadi4Mat or locally, with a desktop application. The process manager provides an interface for executing workflows and communicates on behalf of the user with multiple process engines, to which the actual execution of workflows is delegated. The engines are responsible for the actual processing of the different steps, based on the information defined in a workflow. Data and metadata can either be stored externally or locally.\n\n\n\nSince the reproducibility of the performed steps is a key objective of the workflows, all meaningful information and metadata can be logged along the way. The logging needs to be flexible in order to accommodate different individual or organizational needs, and as such, it is also part of the workflow itself. Workflows can also be shared with other users, for example, via Kadi4Mat. Manual steps may require interaction during the execution of a workflow, for which the system must prompt the user. In summary, the focus of the ELN component thus points in a different direction than in classic ELNs, with the emphasis on the automation of the steps performed. This aspect in particular is similar to systems such as Galaxy[17], which focuses on computational biology, or Taverna[33], a dedicated workflow management system. Nevertheless, some typical features of classic ELNs are also considered in the ELN component, such as the inclusion of handwritten notes.\n\nRepository \nIn the repository component, data management is regarded as the central element, especially the structured data storage and exchange. An important aspect is the enrichment of data with corresponding descriptive metadata, which is required for its description, analysis, or search. Many repositories, especially those focused on publishing research data, use the metadata schema provided by DataCite[34] and are either directly or heavily based on it. This schema is widely supported and enables the direct publication of data via the corresponding DataCite service. For use cases that go beyond data publications, it is limited in its descriptive power, at the same time. There are comparatively few subject-specific schemas available for engineering and material sciences. Two examples are EngMeta[35] and NOMAD Meta Info.[36] The first schema is created a priori and aims to provide a generic description of computer-aided engineering data, while the second schema is created a posteriori, using existing computing inputs and outputs from the database of the NOMAD repository.\nThe second approach is also pursued in a similar way in Kadi4Mat. Instead of a fixed metadata schema, the concrete structure is largely determined by the users themselves, and thus is oriented towards their specific needs. To aid with establishing common metadata vocabularies, a mechanism to create templates is provided. Templates can impose certain restrictions and validations on certain metadata. They are user-defined and can be shared within workgroups or projects, facilitating the establishment of metadata standards. Nevertheless, individual, generic metadata fields, such as a title or description of a data set, can be static. For different use cases such as data analysis, publishing, or the interoperability with other systems, additional conversions must be provided. This is not only necessary because of differing data formats, but also to map vocabularies of different schemas accordingly. Such converted metadata can either represent a subset of existing schemas or require additional fields, such as a license for the re-use of published data. In the long run, the objective in Kadi4Mat is to offer well-defined structures and semantics by making use of ontologies. In the field of materials science, there are ongoing developments in this respect, such as the European Materials Modelling Ontology.[37] However, a bottom-up procedure is considered as a more flexible solution, with the objective to generate an ontology from existing metadata and relationships between different data sets. Such a two-pronged approach aims to be functional in the short term, while still staying extensible in the long term[38], although it heavily depends on how users manage their data and metadata with the options available.\nIn addition to the metadata, the actual data must be managed as well. Here, one can distinguish between data managed directly by Kadi4Mat and linked data. In the simplest form, the former resides on a file system accessible by the repository, which means full control over the data. This requires a copy of each datum to be made available in Kadi4Mat, which makes it less suitable for very large amounts of data. The same applies to data analyses that are to be carried out on external computing infrastructures and must access the data for this purpose. Linked data, on the other hand, can be located on external data storage devices, e.g., high-performance computing infrastructures. This also makes it possible to integrate existing infrastructures and repositories. In these cases, Kadi4Mat can simply offer a view on top of such infrastructures or a more direct integration, depending on the concrete system in question.\nA further point to be addressed within the repository is the publication of data and metadata, including templates and workflows, that require persistent identifiers to be referenceable. Many existing repositories and systems are already specialized in exactly this use case and offer infrastructures for the long-term archiving of large amounts of data. Thus, an integration of suitable external systems is to be considered for this task in particular. From Kadi4Mat\u2019s point of view, only certain basic requirements have to be ensured in order to enable the publishing of data. These include the assignment of a unique identifier within the system, the provision of metadata and licenses, necessary for a publication, and a basic form of user-guided quality control. The repository component thus also goes in a different direction than classic repositories. In a typical scientific workflow, it is primarily focused on all steps that take place between the initial data acquisition and the publishing of data. The component is therefore best described as a community repository that manages warm data (i.e., unpublished data that needs further analysis) and enables data exchange within specific communities, e.g., within a research group or project.\n\nImplementation \nKadi4Mat is built as a web-based application that employs a classic client-server architecture. A graphical front end is provided to be used with a normal web browser as a client, while the server is responsible for handling the back end and the integration of external systems. A high-level overview of the implementation is shown in Figure 3. The front end is based on the classic web technologies JavaScript, HTML, and CSS. In particular, the client-side JavaScript web framework Vue.js[39] is used. The framework is especially suitable for the creation of single-page web applications (SPA), but it can also be used for individual sections of more classic applications, to incrementally add complex and dynamic user interface components to certain pages. Vue.js is mainly used for the latter, the benefit being a clear separation between the data and the presentation layer, as well as the easier re-use of user interface components. This aspect is combined with server-side rendering. Due to the technologies and standards employed, the use of the front end is currently limited to recent versions of modern web browsers such as Firefox, Chrome, or Edge.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Overview of the implementation of Kadi4Mat, separated into front end and back end. The front end uses classic web technologies and is usually operated via a web browser. In the back end, the functionality is split into the web and the core component. The former takes care of the external interfaces, while the latter contains most of the core functionality and handles the interfaces of other systems. A plugin component is also shown, which can be used to customize or extend the functionality of the system.\n\n\n\nIn the back end, the framework Flask[40] is used for the web component. The framework is implemented in Python and is compatible with the common web server gateway interface (WSGI), which specifies an interface between web servers and Python applications. As a so-called microframework, the functionality of Flask itself is limited to the basic features. This means that most of the functionality, which is unrelated to the web component, has to be added by custom code or suitable libraries. At the same time, more freedom is offered in the concrete choice of technologies. This is in direct contrast to web frameworks such as Django[41], which already provides a lot of functionality from scratch. The web component itself is responsible for handling client requests for specific endpoints and assigning them to the appropriate Python functions. Currently, either HTML or JSON is returned, depending on the endpoint. The latter is used as part of an HTTP application programming interface (API), to enable an internal and external programmatic data exchange. This API is based on the representational state transfer (REST) paradigm.[42] Support for other exchange formats could also be relevant in the future, particularly for implementing certain exchange formats for interoperability, such as OAI-PMH.[43] Especially for handling larger amounts of data, other exchange protocols besides HTTP are considered.\nA large part of the application consists of the core functionality, which is divided into different modules, as shown in Figure 3. This structure is mainly of an organizational nature. A microservice architecture is currently not implemented. Modules that access external components are particularly noteworthy, which is an aspect that will also be increasingly important in the future. External components can either run on the same hardware as Kadi4Mat itself or on separate systems available via a network interface. For the storage of metadata, the persistence module makes use of the relational database management system PostgreSQL[44], while the regular file system stores the actual data. Additionally, the software Elasticsearch[45] is used to index all the metadata that needs to be efficiently searchable. The aforementioned process manager[46], which is currently implemented as a command line application, manages the execution of workflows by delegating each execution task to an available process engine.[47] While the current implementation of the process engine primarily uses the local file system of the machine on which it is running, users can add steps to synchronize data with the repository to their workflow at will. To increase performance with multiple parallel requests for workflow execution, the requests can be distributed to process engines running on additional servers. By wrapping the process manager with a simple HTTP API, for example, its interface can easily be used over a network. A message broker is used to decouple longer running or periodically executed background tasks from the rest of the application, by delegating them to one or more background worker processes. Apart from using locally managed user accounts or an LDAP system for authentication, Shibboleth[48] can be used as well. From a technical point of view, Shibboleth is not a single system, but the interaction of several components, which together enable a distributed authentication procedure. Depending on the type of authentication, user attributes or group affiliations can also be used for authorization purposes in the future.\nAnother component shown in Figure 3 involves the plugins. These can be used to customize or extend the basic functionality of certain procedures or actions, without having to modify or know the corresponding implementation in detail. Unlike the tools in a workflow, plugins make use of predefined hooks to add their custom functionality. While such a plugin has to be installed centrally by the system administrator for all users of a Kadi4Mat instance, the possibilities are also evaluated to be able to make use of individual plugins on the user level.\n\nResults \nThe current functionalities of Kadi4Mat can either be utilized via the graphical user interface, with a browser, or via the HTTP API, with a suitable client. On top of the API, a Python library is developed, which makes it especially easy to interact with the different functionalities.[49] Besides using the library in Python code, it offers a command-line interface, enabling the integration with other programming or scripting languages.\nIn the following, the most important features of Kadi4Mat are explained, based on its graphical user interface. The focus of the features implemented so far is on the repository component, the topics of structured data management and data exchange in particular, as well as on the workflows, which are a central part of the ELN\u2019s functionality. After logging in to Kadi4Mat, it is possible to create different types of resources. The most important type of resource are the so-called records, which can link arbitrary data with descriptive metadata and serve as basic components that can be used in workflows and future data publications. In principle, a record can be used for all kinds of data, including data from simulations or experiments, and it can be linked to other records of related data sets, e.g., to the descriptions of the software and hardware devices used. \nThe metadata of a record includes both basic metadata, such as title or description, and domain-specific metadata, which can be specified generically, in the form of key\/value pairs. The latter can be defined using a special editor, as shown in Figure 4. With the help of such metadata, a description of subject- and application-specific records becomes possible. This is particularly relevant in an interdisciplinary research field such as materials science, where using a fixed schema would be impracticable, due to the heterogeneity of the data formats and the corresponding metadata. The value of each metadata entry can be of different types, such as simple character strings or numeric types like integers and floating point numbers. Numeric values can also be provided with an arbitrary unit. Furthermore, nested types can be used to represent metadata structures of almost any complexity, for example in the form of lists. The input of such structures can be simplified by templates, which are specified in advance and can be combined as desired. While templates currently offer the same possibilities as the actual metadata, it is planned to add further validation functionalities, such as the specification of a selection of valid values for certain metadata keys. Wherever possible, automatically recorded metadata is also available in each record, such as the creator of the record or the creation date. The actual data of the record can be uploaded by the users and are currently stored on a file system, accessible by Kadi4Mat. It is possible to upload any number of files for each record. This can be helpful when dealing with a series of several hundred images of a simulated microstructure, for example, which all share the same metadata.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Screenshot of the generic metadata editor, showing the different types of metadata entries currently possible. The last two examples of type dictionary and list contain nested metadata entries. In the upper right corner, a menu is displayed that allows performing various actions, one of which switches to a tree-based overview of the metadata. The ability to select metadata templates is shown in the lower right corner.\n\n\n\nThe created record can be viewed on an overview page that displays all metadata and linked files. Some common file formats include a preview that is directly integrated into the web browser, such as image files, PDFs, archives, or textual data. Furthermore, the access rights of the record are displayed on its overview page. Currently, two levels of visibility can be set when creating a record: public and private visibility. While public records can be viewed by every logged-in user\u2014i.e., read rights are granted implicitly to each user\u2014private records can initially only be viewed by their creator. Only the creator of a record can perform further actions, such as editing the metadata or deleting the entire record. Figure 5 shows the overview page of a record, including its metadata and the menu to perform the previously mentioned actions. \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Screenshot of a record overview page. The basic metadata is shown, followed by the generic metadata entries (shown as extra metadata). The menu on the top allows various actions to be performed on the current record. The tabs below the menu are used to switch to other views that display the files and other resources associated with the current record, as well as access permissions and a history of metadata revisions.\n\n\n\nIn order to grant different access rights to other users, even within private records, different roles can be defined for any user in a separate view. Currently, the roles are static, which means that they can be selected from a predefined list and are each linked to the corresponding fine-grained permissions. Because of these permissions, the possibility of custom roles or certain actions being linked to different user attributes becomes possible. In addition to roles for individual users, roles can also be defined for groups. These are simple groupings of several users which, similar to records, can be created and managed by the users themselves. The same roles that can be defined for individual users can be assigned to groups as well. Each member of the group is granted the corresponding access rights automatically. \nFinally, the overview page of a record also shows the resources linked to it. This refers in particular to the so-called collections. Collections represent simple, logical groupings of multiple records and can thus contribute to a better organization of resources. In terms of an ontology, collections can be regarded as classes, while records inside a collection represent concrete instances of such a class. Like records and groups, collections can be created and managed by users. Records can also be linked to other records. Each record link represents a separate resource, which in turn can contain certain metadata. The ability to specify generic metadata and such resource links already enables a basic ontology-like structure. This structure can be further improved in the future, e.g., by using different types of links, with varying semantics, and by allowing collections to be nested.\nTo be able to find different resources efficiently, especially records, a search function is included in Kadi4Mat. This allows searching in the basic metadata of resources and in the generic metadata of records via keywords or full text search. The values of nested generic metadata entries are flattened before they are indexed in ElasticSearch. This way, a common search mapping can be defined for all kinds of generic metadata. The search results can be sorted and filtered in various ways, for example, by using different user-defined tags or data formats, in the case of records. Figure 6 shows an example search of records, with the corresponding results.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. Screenshot of the search functionality of records with the corresponding search results. In addition to providing a simple query for searching the basic metadata of a record, the generic metadata can also be searched by specifying desired keys, types, or values. The searchable types are derived from the actual types of the generic metadata entries, e.g., integers and floating point numbers are grouped together as numeric type. Various other options are offered for filtering and sorting the search results.\n\n\n\nWhile the execution of workflows, via the web interfaces, and the ability to add user-defined tools are still under development, it is possible to define a workflow using a graphical node editor, running in the web browser. Figure 7 shows a simple example workflow created with this editor. A selection of predefined nodes can be combined and parameterized, while the resulting workflow can be downloaded. A custom JSON-based format is currently used to store the representation of a workflow. This format contains all the information for the node editor to correctly display the workflow and to derive a functional workflow representation for execution. The downloaded workflow file can be executed directly on a local workstation by using the command line interface of the process manager. All tools to actually run such a workflow need to be installed beforehand. A selection of tools is provided for various tasks[50], including connecting to a Kadi4Mat instance by using a suitable wrapper on top of the aforementioned API library. Several common use cases have already been implemented, including the task of extracting metadata from Excel spreadsheets, often used to replace an actual ELN system, and importing it into Kadi4Mat. An overview of such a workflow is shown in Figure 8. Developments are also underway for data science applications, especially in the field of machine learning. The combination of the ELN and the repository fits particularly well with the requirements of such applications, which typically require lots of high-quality input data to function well.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 7. Screenshot of a workflow created with the web-based node editor. Several String input nodes are shown, as well as a special node that prompts the user to enter a file (UserInput: File). The two tools mkdir and ImageJMacro are used to create a new directory and to execute an ImageJ[51] macro file, respectively. The latter uses the input file the user was asked for. Except for the input nodes, all nodes are connected via an explicit dependency.\n\n\n\n\n\n\n\n\n\n\n\n\n Figure 8. Overview of an exemplary workflow using Kadi4Mat. The starting point is raw data and corresponding metadata stored in an Excel spreadsheet. The tools used in this workflow are divided into tools for data handling and tools for data transport, the latter referring to the Kadi4Mat integration. In a first conversion step, the metadata are transformed into a format readable by the API of Kadi4Mat and linked to the raw data by creating a new record. The raw data is further analyzed using the metadata stored in Kadi4Mat. Finally, the result of the analysis is plotted and both data sets are uploaded to Kadi4Mat as records. All records can be linked to each other in a further step, either as part of the workflow or separately.\n\n\n\nConclusion \nThe development and current functionality of the research data infrastructure Kadi4Mat is presented. The objective of this infrastructure is to combine the features of an ELN and a repository in such a way that researchers can be supported throughout the whole research process. The ongoing development aims at covering the heterogeneous use cases of materials science disciplines. For this purpose, flexible metadata schemas, workflows, and tools are especially important, as is the use of custom installations and instances. The basic functionality of the repository component is largely given by the features already implemented and can be used with a graphical as well as a programmatic interface. This includes, above all, uploading, managing, and exchanging data, as well as the associated metadata. The latter can be defined with a flexible metadata editor to accommodate the needs of different users and workgroups. A search functionality enables the efficient retrieval of the data. The essential infrastructure for workflows is implemented as a central part of the ELN component. Simple workflows can be defined with an initial version of the web-based node editor and executed locally using provided tools and the process manager\u2019s command-line interface. Both main components are improved continuously. \nVarious other features that have not yet been mentioned as part of the concept are planned or are already in the conception stage. These include the optional connection of several Kadi4Mat instances, a more direct, low-level access to data, and the integration of an app store, for the central administration of tools and plugins.\nThe development of Kadi4Mat largely follows a bottom-up approach. Instead of developing concepts in advance, to cover as many use cases as possible, a basic technical infrastructure is established first. On this basis, further steps are evaluated in exchange with interested users and by implementing best practice examples. Due to the heterogeneous nature of materials science, most features are kept very generic. The concrete structuring of the data storage, the metadata, and the workflows is largely left to the users. As a positive side effect, an extension of the research data infrastructure to other disciplines is possible in the future.\n\nAcknowledgements \nThis work is supported by the Federal Ministry of Education and Research (BMBF) in the projects FestBatt (project number 03XP0174E) and as part of the Excellence Strategy of the German Federal and State Governments, by the German Research Foundation (DFG) in the projects POLiS (project number 390874152) and SuLMaSS (project number 391128822) and by the Ministry of Science, Research and Art Baden-W\u00fcrttemberg in the project MoMaF \u2013 Science Data Center, with funds from the state digitization strategy digital@bw (project number 57). The authors are also grateful for the editorial support of Leon Geisen.\n\nCompeting interests \nThe authors have no competing interests to declare.\n\nReferences \n\n\n\u2191 Sandfeld, S.; Dahmen, T.; Fischer, F.O.R. et al. (2018). \"Strategiepapier - Digitale Transformation in der Materialwissenschaft und Werkstofftechnik\". Deutsche Gesellschaft f\u00fcr Materialkunde e.V. https:\/\/www.tib.eu\/en\/search\/id\/TIBKAT%3A1028913559\/ .   \n\n\u2191 Hey, T.; Trefethen, A. (2003). \"Chapter 36: The Data Deluge: An e\u2010Science Perspective\". In Berman, F.; Fox, G.; Hey, T.. Grid Computing: Making the Global Infrastructure a Reality. John Wiley & Sons, Ltd. doi:10.1002\/0470867167.ch36. ISBN 9780470867167.   \n\n\u2191 Hey, T.; Tansley, S.; Tolle, K. (2009). The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research. ISBN 9780982544204. https:\/\/www.microsoft.com\/en-us\/research\/publication\/fourth-paradigm-data-intensive-scientific-discovery\/ .   \n\n\u2191 4.0 4.1 Hill, J.; Mulholland, G.; Persson, K. et al. (2016). \"Materials science with large-scale data and informatics: Unlocking new opportunities\". MRS Bulletin 41 (5): 399\u2013409. doi:10.1557\/mrs.2016.93.   \n\n\u2191 Naughton, L.; Kernohan, D. (2016). \"Making sense of journal research data policies\". Insights 29 (1): 84\u20139. doi:10.1629\/uksg.284.   \n\n\u2191 Pampel, H.; Vierkant, P.; Scholze, F. et al. (2013). \"Making Research Data Repositories Visible: The re3data.org Registry\". PLoS One 8 (11): e78080. doi:10.1371\/journal.pone.0078080.   \n\n\u2191 Sansone, S.-A.; McQuilton, P.; Rocca-Serra, P. et al. (2019). \"FAIRsharing as a community approach to standards, repositories and policies\". Nature Biotechnology 37: 358\u201367. doi:10.1038\/s41587-019-0080-8.   \n\n\u2191 Jain, A.; Ong, S.P.; Hautier, G. et al. (2013). \"Commentary: The Materials Project: A materials genome approach to accelerating materials innovation\". APL Materials 1 (1): 011002. doi:10.1063\/1.4812323.   \n\n\u2191 Draxl, C.; Scheffler, M. (2018). \"NOMAD: The FAIR concept for big data-driven materials science\". MRS Bulletin 43 (9): 676\u201382. doi:10.1557\/mrs.2018.208.   \n\n\u2191 \"CKAN\". CKAN Association. https:\/\/ckan.org\/ . Retrieved 19 May 2020 .   \n\n\u2191 King, G. (2007). \"An Introduction to the Dataverse Network as an Infrastructure for Data Sharing\". Sociological Methods & Research 36 (2): 173\u201399. doi:10.1177\/0049124107306660.   \n\n\u2191 Smith, M.; Barton, M.; Bass, M. et al. (2003). \"DSpace: An Open Source Dynamic Digital Repository\". D-Lib Magazine 9 (1). doi:10.1045\/january2003-smith.   \n\n\u2191 \"Invenio\". CERN. https:\/\/invenio-software.org\/ . Retrieved 19 May 2020 .   \n\n\u2191 European Organization for Nuclear Research (2013). \"Zenodo\". CERN. doi:10.25495\/7GXK-RD71. https:\/\/www.zenodo.org\/ .   \n\n\u2191 Amorim, R.C.; Castro, J.A.; da Silva, J.R. et al. (2017). \"A comparison of research data management platforms: Architecture, flexible metadata and interoperability\". Universal Access in the Information Society 16: 851\u201362. doi:10.1007\/s10209-016-0475-y.   \n\n\u2191 Rubacha, M.; Rattan, A.K.; Hosselet, S.C. (2011). \"A Review of Electronic Laboratory Notebooks Available in the Market Today\". SLAS Technology 16 (1). doi:10.1016\/j.jala.2009.01.002.   \n\n\u2191 17.0 17.1 Afgan, E.; Baker, D.; Batut, B. et al. (2018). \"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2018 update\". Nucleic Acids Research 46 (W1). doi:10.1093\/nar\/gky379.   \n\n\u2191 Kluyver, T.; Ragan-Kelley, B.; P\u00e9rez, F. et al. (2016). \"Jupyter Notebooks\u2014A publishing format for reproducible computational workflows\". In Loizides, F.; Schmidt, B.. Positioning and Power in Academic Publishing: Players, Agents and Agendas. IOS Press. pp. 87\u201390. doi:10.3233\/978-1-61499-649-1-87.   \n\n\u2191 Bird, C.L.; Willoughby, C.; Frey, J.G. (2013). \"Laboratory notebooks in the digital era: The role of ELNs in record keeping for chemistry and other sciences\". Chemical Society Reviews 42 (20): 8157\u20138175. doi:10.1039\/C3CS60122F.   \n\n\u2191 Elliott, M.H. (2009). \"Thinking Beyond ELN\". Scientific Computing 26 (6): 6\u201310. Archived from the original on 20 May 2011. https:\/\/web.archive.org\/web\/20110520065023\/http:\/\/www.scientificcomputing.com\/articles-IN-Thinking-Beyond-ELN-120809.aspx .   \n\n\u2191 21.0 21.1 Taylor, K.T. (2006). \"The status of electronic laboratory notebooks for chemistry and biology\". Current Opinion in Drug Discovery and Development 9 (3): 348\u201353. PMID 16729731.   \n\n\u2191 Carpi, N.; Minges, A.; Piel, M. (2017). \"eLabFTW: An open source laboratory notebook for research labs\". Journal of Open Source Software 2 (12): 146. doi:10.21105\/joss.00146.   \n\n\u2191 \"SciNote\". SciNote LLC. https:\/\/www.scinote.net\/ . Retrieved 21 May 2020 .   \n\n\u2191 Tremouilhac, P.; Nguyen, A.; Huang, Y.-C. et al. (2017). \"Chemotion ELN: An open source electronic lab notebook for chemists in academia\". Journal of Cheminformatics 9: 54. doi:10.1186\/s13321-017-0240-0.   \n\n\u2191 Heidorn, P.B. (2008). \"Shedding Light on the Dark Data in the Long Tail of Science\". Library Trends 57 (2): 280\u201399. doi:10.1353\/lib.0.0036.   \n\n\u2191 Carusi, A.; Reimer, T.F. (17 January 2010). \"Virtual Research Environment Collaborative Landscape Study\". JISC. doi:10.25561\/18568. http:\/\/hdl.handle.net\/10044\/1\/18568 .   \n\n\u2191 Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J. et al. (2016). \"The FAIR Guiding Principles for scientific data management and stewardship\". Scientific Data 3: 160018. doi:10.1038\/sdata.2016.18. PMC PMC4792175. PMID 26978244. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175 .   \n\n\u2191 \"Kadi4Mat\". Karlsruhe Institute of Technology. https:\/\/kadi.iam-cms.kit.edu\/ . Retrieved 30 September 2020 .   \n\n\u2191 Brandt, N.; Griem, L.; Hermann, C. et al. (2020). \"IAM-CMS\/kadi: Kadi4Mat (Version 0.6.0)\". Zenodo. doi:10.5281\/zenodo.4507826.   \n\n\u2191 \"MATLAB\". MathWorks. https:\/\/www.mathworks.com\/products\/matlab.html . Retrieved 19 January 2021 .   \n\n\u2191 Hawker, C.D. (2007). \"Laboratory Automation: Total and Subtotal\". Clinics in Laboratory Medicine 27 (4): 749\u201370. doi:10.1016\/j.cll.2007.07.010.   \n\n\u2191 Potthoff, J.; Tremouilhac, P.; Hodapp, P. et al. (2019). \"Procedures for systematic capture and management of analytical data in academia\". Analytica Chimica Acta: X 1: 100007. doi:10.1016\/j.acax.2019.100007.   \n\n\u2191 Wolstencroft, K.; Haines, R.; Fellows, D. et al. (2013). \"The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud\". Nucleic Acids Research 41 (W1): W557\u2013W561. doi:10.1093\/nar\/gkt328.   \n\n\u2191 DataCite Metadata Working Group (16 August 2019). \"DataCite Metadata Schema for the Publication and Citation of Research Data. Version 4.3\". DataCite e.V. https:\/\/schema.datacite.org\/meta\/kernel-4.3\/ .   \n\n\u2191 Schembera, B.; Iglezakis, D. (2020). \"EngMeta: Metadata for computational engineering\". International Journal of Metadata, Semantics and Ontologies 14 (1): 26\u201338. doi:10.1504\/IJMSO.2020.107792.   \n\n\u2191 Ghiringhelli, L.M.; Carbogno, C.; Levchenko, S. et al. (2017). \"Towards efficient data exchange and sharing for big-data driven materials science: metadata and data formats\". Computational Materials 3: 46. doi:10.1038\/s41524-017-0048-5.   \n\n\u2191 European Materials Modelling Council. \"European Materials and Modelling Ontology\". GitHub. https:\/\/github.com\/emmo-repo . Retrieved 24 May 2020 .   \n\n\u2191 Greenberg, J.; White, H.C.; Carrier, S. et al. (2009). \"A Metadata Best Practice for a Scientific Data Repository\". Journal of Library Metadata 9 (3\u20134): 194\u2013212. doi:10.1080\/19386380903405090.   \n\n\u2191 Vue Core Development Team. \"Vue.js - The Progressive JavaScript Framework\". Evan You. https:\/\/vuejs.org\/ . Retrieved 25 May 2020 .   \n\n\u2191 \"Flask\". The Pallets Project. https:\/\/palletsprojects.com\/p\/flask\/ . Retrieved 25 May 2020 .   \n\n\u2191 \"Django\". The Django Software Foundation. https:\/\/www.djangoproject.com\/ . Retrieved 25 May 2020 .   \n\n\u2191 Fielding, R.T. (2000). \"Architectural Styles and the Design of Network-based Software Architectures\". University of California, Irvine. https:\/\/www.ics.uci.edu\/~fielding\/pubs\/dissertation\/top.htm .   \n\n\u2191 \"Open Archives Initiative Protocol for Metadata Harvesting\". Open Archives Initiative. http:\/\/www.openarchives.org\/pmh\/ . Retrieved 25 May 2020 .   \n\n\u2191 \"PostegreSQL: The World's Most Advanced Open Source Relational Database\". PostgreSQL Global Development Group. https:\/\/www.postgresql.org\/ . Retrieved 30 September 2020 .   \n\n\u2191 \"Elasticsearch: The Heart of the Free and Open Elastic Stack\". Elasticsearch B.V. https:\/\/www.elastic.co\/elasticsearch\/ . Retrieved 02 June 2020 .   \n\n\u2191 Zschumme, P. (15 January 2021). \"IAM-CMS\/process-manager (Version 0.1.0)\". Zenodo. doi:10.5281\/zenodo.4442553. https:\/\/zenodo.org\/record\/4442553 .   \n\n\u2191 Zschumme, P. (15 January 2021). \"IAM-CMS\/process-engine (Version 0.1.0)\". Zenodo. doi:10.5281\/zenodo.4442563. https:\/\/zenodo.org\/record\/4442563 .   \n\n\u2191 Walker, D.; Cantor, S.; Carmody, S.; et al. (17 November 2017). \"Shibboleth Architecture Protocols and Profiles\". Internet2 Confluence. doi:http:\/\/doi.org\/10.26869\/TI.66.1. https:\/\/spaces.at.internet2.edu\/display\/TI\/TI.66.1 .   \n\n\u2191 Schoof, E.; Brandt, N. (02 February 2021). \"IAM-CMS\/kadi-apy: Kadi4Mat API Library (Version 0.4.1)\". Zenodo. doi:10.5281\/zenodo.4507865. https:\/\/zenodo.org\/record\/4088276 .   \n\n\u2191 Zschumme, P. (16 October 2020). \"IAM-CMS\/workflow-nodes (Version 0.1.0)\". Zenodo. doi:10.5281\/zenodo.4094719. https:\/\/zenodo.org\/record\/4094719 .   \n\n\u2191 Schindelin, J.; Rueden, C.T.; Hiner, M.C. et al. (2015). \"The ImageJ ecosystem: An open platform for biomedical image analysis\". Molecular Reproduction & Development 82 (7\u20138): 518\u201329. doi:10.1002\/mrd.22489.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; however, this version lists them in order of appearance, by design.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\">https:\/\/www.limswiki.org\/index.php\/Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on data management and sharingLIMSwiki journal articles on materials informaticsLIMSwiki journal articles on research\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 24 February 2021, at 20:11.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 569 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","7d7600ccadb42ccd7bdbe90737c9955d_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Kadi4Mat_A_research_data_infrastructure_for_materials_science skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Kadi4Mat: A research data infrastructure for materials science<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>The concepts and current developments of a research data infrastructure for <a href=\"https:\/\/www.limswiki.org\/index.php\/Materials_informatics\" title=\"Materials informatics\" class=\"wiki-link\" data-key=\"4dd2125beb9794d0a679b921981f1ddc\">materials science<\/a> are presented, extending and combining the features of an <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_laboratory_notebook\" title=\"Electronic laboratory notebook\" class=\"wiki-link\" data-key=\"a9fbbd5e0807980106763fab31f1e72f\">electronic laboratory notebook<\/a> (ELN) and a repository. The objective of this infrastructure is to incorporate the possibility of structured data storage and data exchange with documented and reproducible <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">data analysis<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_visualization\" title=\"Data visualization\" class=\"wiki-link\" data-key=\"4a3b86cba74bc7bb7471aa3fc2fcccc3\">visualization<\/a>, which finally leads to the publication of the data. This way, researchers can be supported throughout the entire research process. The software is being developed as a web-based and desktop-based system, offering both a graphical user interface (GUI) and a programmatic interface. The focus of the development is on the integration of technologies and systems based on both established as well as new concepts. Due to the heterogeneous nature of materials science data, the current features are kept mostly generic, and the structuring of the data is largely left to the users. As a result, an extension of the research data infrastructure to other disciplines is possible in the future. The source code of the project is publicly available under a permissive Apache 2.0 license.\n<\/p><p><b>Keywords<\/b>: research data management, electronic laboratory notebook, repository, open source, materials science\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>In the engineering sciences, the handling of digital research data plays an increasingly important role in all fields of application.<sup id=\"rdp-ebb-cite_ref-SandfeldStrateg18_1-0\" class=\"reference\"><a href=\"#cite_note-SandfeldStrateg18-1\">[1]<\/a><\/sup> This is especially the case, due to the growing amount of data obtained from experiments and simulations.<sup id=\"rdp-ebb-cite_ref-HeyTheData03_2-0\" class=\"reference\"><a href=\"#cite_note-HeyTheData03-2\">[2]<\/a><\/sup> The extraction of knowledge from these data is referred to as a data-driven, fourth paradigm of science, filed under the keyword \"data science.\"<sup id=\"rdp-ebb-cite_ref-HeyTheFourth09_3-0\" class=\"reference\"><a href=\"#cite_note-HeyTheFourth09-3\">[3]<\/a><\/sup> This is particularly true in <a href=\"https:\/\/www.limswiki.org\/index.php\/Materials_informatics\" title=\"Materials informatics\" class=\"wiki-link\" data-key=\"4dd2125beb9794d0a679b921981f1ddc\">materials science<\/a>, as the research and understanding of new materials are becoming more and more complex.<sup id=\"rdp-ebb-cite_ref-HillMaterials16_4-0\" class=\"reference\"><a href=\"#cite_note-HillMaterials16-4\">[4]<\/a><\/sup> Without suitable <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analysis<\/a> methods, the ever-growing amount of data will no longer be manageable. In order to be able to perform appropriate data analyses smoothly, the structured storage of research data and associated <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a> is an important aspect. Specifically, a uniform research <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">data management<\/a> is needed, which is made possible by appropriate infrastructures such as research data repositories. In addition to uniform data storage, such systems can help to overcome inter-institutional hurdles in data exchange, compare theoretical and experimental data, and provide reproducible <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a> for data analysis. Furthermore, linking the data with persistent identifiers enables other researchers to directly reference them in their work.\n<\/p><p>In particular, repositories for the storage and internal or public exchange of research data are becoming increasingly widespread. In particular, the publication of such data, either on its own or as a supplement to a text publication, is increasingly encouraged or sometimes even required.<sup id=\"rdp-ebb-cite_ref-NaughtonMaking16_5-0\" class=\"reference\"><a href=\"#cite_note-NaughtonMaking16-5\">[5]<\/a><\/sup> In order to find a suitable repository, services such as re3data.org<sup id=\"rdp-ebb-cite_ref-PampelMaking13_6-0\" class=\"reference\"><a href=\"#cite_note-PampelMaking13-6\">[6]<\/a><\/sup> or FAIRSharing<sup id=\"rdp-ebb-cite_ref-SansoneFAIR19_7-0\" class=\"reference\"><a href=\"#cite_note-SansoneFAIR19-7\">[7]<\/a><\/sup> are available. These services also make it possible to find subject-specific repositories for materials science data. Two well-known examples are the Materials Project<sup id=\"rdp-ebb-cite_ref-JainComment13_8-0\" class=\"reference\"><a href=\"#cite_note-JainComment13-8\">[8]<\/a><\/sup> and the NOMAD Repository.<sup id=\"rdp-ebb-cite_ref-DraxlNOMAD18_9-0\" class=\"reference\"><a href=\"#cite_note-DraxlNOMAD18-9\">[9]<\/a><\/sup> Indexed repositories are usually hosted centrally or institutionally and are mostly used for the publication of data. However, some of the underlying systems can also be installed by the user, e.g., for internal use within individual research groups. Additionally, this allows full control over stored data as well as internal data exchanges, if this function is not already part of the repository. In this respect, open-source systems are particularly important, as this means independence from vendors and opens up the possibility of modifying the existing functionality or adding additional features, sometimes via built-in plug-in systems. Examples of such systems are CKAN<sup id=\"rdp-ebb-cite_ref-CKANHome_10-0\" class=\"reference\"><a href=\"#cite_note-CKANHome-10\">[10]<\/a><\/sup>, Dataverse<sup id=\"rdp-ebb-cite_ref-KinganIntro07_11-0\" class=\"reference\"><a href=\"#cite_note-KinganIntro07-11\">[11]<\/a><\/sup>, <a href=\"https:\/\/www.limswiki.org\/index.php\/DSpace\" title=\"DSpace\" class=\"wiki-link\" data-key=\"a722996d9e71fde809676b66f183df91\">DSpace<\/a><sup id=\"rdp-ebb-cite_ref-SmithDSpace03_12-0\" class=\"reference\"><a href=\"#cite_note-SmithDSpace03-12\">[12]<\/a><\/sup>, and Invenio<sup id=\"rdp-ebb-cite_ref-InvenioHome_13-0\" class=\"reference\"><a href=\"#cite_note-InvenioHome-13\">[13]<\/a><\/sup>, where the latter is the basis of Zenodo.<sup id=\"rdp-ebb-cite_ref-ZenodoHome_14-0\" class=\"reference\"><a href=\"#cite_note-ZenodoHome-14\">[14]<\/a><\/sup> The listed repositories are all generic and represent only a selection of the existing open-source systems.<sup id=\"rdp-ebb-cite_ref-AmorimAComp16_15-0\" class=\"reference\"><a href=\"#cite_note-AmorimAComp16-15\">[15]<\/a><\/sup>\n<\/p><p>In addition to repositories, a second type of system increasingly being used in experiment-oriented research areas is the <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_laboratory_notebook\" title=\"Electronic laboratory notebook\" class=\"wiki-link\" data-key=\"a9fbbd5e0807980106763fab31f1e72f\">electronic laboratory notebook<\/a> (ELN).<sup id=\"rdp-ebb-cite_ref-RubachaARev11_16-0\" class=\"reference\"><a href=\"#cite_note-RubachaARev11-16\">[16]<\/a><\/sup> Nowadays, the functionality of ELNs goes far beyond the simple replacement of paper-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_notebook\" title=\"Laboratory notebook\" class=\"wiki-link\" data-key=\"be60c7be96aba8e9a84537fd8835fa54\">laboratory notebooks<\/a>, and can also include aspects such as data analysis, as seen, for example, in <a href=\"https:\/\/www.limswiki.org\/index.php\/Galaxy_(biomedical_software)\" title=\"Galaxy (biomedical software)\" class=\"wiki-link\" data-key=\"ead5d6ebaa8d67744d2f68d454d89ce6\">Galaxy<\/a><sup id=\"rdp-ebb-cite_ref-AfganTheGal18_17-0\" class=\"reference\"><a href=\"#cite_note-AfganTheGal18-17\">[17]<\/a><\/sup> or <a href=\"https:\/\/www.limswiki.org\/index.php\/Jupyter_Notebook\" title=\"Jupyter Notebook\" class=\"wiki-link\" data-key=\"26fd35430c10e009a142bbab5dbf617a\">Jupyter Notebook<\/a>.<sup id=\"rdp-ebb-cite_ref-KluyverJupyter16_18-0\" class=\"reference\"><a href=\"#cite_note-KluyverJupyter16-18\">[18]<\/a><\/sup> Both systems focus primarily on providing accessible and reproducible computational research data. However, the boundary between unstructured and structured data is increasingly becoming blurred, the latter being traditionally only found in <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> (LIMS).<sup id=\"rdp-ebb-cite_ref-BirdLab13_19-0\" class=\"reference\"><a href=\"#cite_note-BirdLab13-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ElliottThink09_20-0\" class=\"reference\"><a href=\"#cite_note-ElliottThink09-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TaylorTheStatus06_21-0\" class=\"reference\"><a href=\"#cite_note-TaylorTheStatus06-21\">[21]<\/a><\/sup> Most existing ELNs are domain-specific and limited to research disciplines such as biology or chemistry.<sup id=\"rdp-ebb-cite_ref-TaylorTheStatus06_21-1\" class=\"reference\"><a href=\"#cite_note-TaylorTheStatus06-21\">[21]<\/a><\/sup> According to current knowledge, a system specifically tailored to materials science does not exist. For ELNs, there are also open-source systems such as <a href=\"https:\/\/www.limswiki.org\/index.php\/ELabFTW\" title=\"ELabFTW\" class=\"wiki-link\" data-key=\"155590ac366985a7ee3cf74ade1df84a\">eLabFTW<\/a><sup id=\"rdp-ebb-cite_ref-CarpiElabFTW17_22-0\" class=\"reference\"><a href=\"#cite_note-CarpiElabFTW17-22\">[22]<\/a><\/sup>, <a href=\"https:\/\/www.limswiki.org\/index.php\/SciNote\" title=\"SciNote\" class=\"wiki-link\" data-key=\"fcafe5e3de96d8ba319b70f089a53ea0\">sciNote<\/a><sup id=\"rdp-ebb-cite_ref-ScinoteHome_23-0\" class=\"reference\"><a href=\"#cite_note-ScinoteHome-23\">[23]<\/a><\/sup>, or <a href=\"https:\/\/www.limswiki.org\/index.php\/Chemotion_ELN\" title=\"Chemotion ELN\" class=\"wiki-link\" data-key=\"69d105f8f7c60427d4fe9a8f21a8019a\">Chemotion<\/a>.<sup id=\"rdp-ebb-cite_ref-TremouilhacChemotionELN17_24-0\" class=\"reference\"><a href=\"#cite_note-TremouilhacChemotionELN17-24\">[24]<\/a><\/sup> Compared to the repositories, however, the selection of ELNs is smaller. Furthermore, only the first two mentioned systems are generic.\n<\/p><p>Thus, generic research data systems and software are available for both ELNs and repositories, which, in principle, could also be used in materials science. The listed open-source solutions are of particular relevance, as they can be adapted to different needs and are generally suitable for use in a custom installation within single research groups. However, both aspects can be a considerable hurdle, especially for smaller groups. Due to a lack of resources, structured research data management and the possibility of making data available for subsequent use is therefore particularly difficult for such groups.<sup id=\"rdp-ebb-cite_ref-HeidornShed08_25-0\" class=\"reference\"><a href=\"#cite_note-HeidornShed08-25\">[25]<\/a><\/sup> What is finally missing is a system that can be deployed and used both centrally and decentrally, as well as internally and publicly, without major obstacles. The system should support researchers throughout the entire research process, starting with the generation and extraction of raw data, up to the structured storage, exchange, and analysis of the data, resulting in the final publication of the corresponding results. In this way, the features of the ELN and the repository are combined, creating a virtual research environment<sup id=\"rdp-ebb-cite_ref-CarusiVirt10_26-0\" class=\"reference\"><a href=\"#cite_note-CarusiVirt10-26\">[26]<\/a><\/sup> that accelerates the generation of innovations by facilitating collaboration among researchers. In an interdisciplinary field like materials science, there is a special need to model the very heterogeneous workflows of researchers.<sup id=\"rdp-ebb-cite_ref-HillMaterials16_4-1\" class=\"reference\"><a href=\"#cite_note-HillMaterials16-4\">[4]<\/a><\/sup>\n<\/p><p>For this purpose, the research data infrastructure Kadi4Mat (Karlsruhe Data Infrastructure for Materials Sciences) is being developed at the Institute for Applied Materials (IAM-CMS) of the Karlsruhe Institute of Technology (KIT). The aim of the software is to combine the possibility of structured data storage with documented and reproducible workflows for data analysis and visualization tasks, incorporating new concepts with established technologies and existing solutions. In the development of the software, the FAIR Guiding Principles<sup id=\"rdp-ebb-cite_ref-WilkinsonTheFAIR16_27-0\" class=\"reference\"><a href=\"#cite_note-WilkinsonTheFAIR16-27\">[27]<\/a><\/sup> for scientific data management are taken into account. Instances of the data infrastructure have already been deployed and today show how structured data storage and data exchange are made possible.<sup id=\"rdp-ebb-cite_ref-Kadi4MatHome_28-0\" class=\"reference\"><a href=\"#cite_note-Kadi4MatHome-28\">[28]<\/a><\/sup> Furthermore, the source code of the project is publicly available under a permissive Apache 2.0 license.<sup id=\"rdp-ebb-cite_ref-BrandtIAM20_29-0\" class=\"reference\"><a href=\"#cite_note-BrandtIAM20-29\">[29]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Concepts\">Concepts<\/span><\/h2>\n<p>Kadi4Mat is logically divided into the two components\u2014an ELN and a repository\u2014which have access to various tools and technical infrastructures. The components can be used by web- and desktop-based applications, via uniform interfaces. Both a graphical and a programmatic interface are provided, using machine-readable formats and various exchange protocols. In Figure 1, a conceptual overview of the infrastructure of Kadi4Mat is presented.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Brandt_DataSciJourn21_20-1.png\" class=\"image wiki-link\" data-key=\"9b91d012e3c6678d8cf37135c4956085\"><img alt=\"Fig1 Brandt DataSciJourn21 20-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/76\/Fig1_Brandt_DataSciJourn21_20-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Conceptual overview of the infrastructure of Kadi4Mat. The system is logically divided into the two components\u2014an ELN and a repository\u2014which have access to various data handling tools and technical infrastructures. The two components can be used both graphically and programmatically via uniform interfaces.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Electronic_laboratory_notebook\">Electronic laboratory notebook<\/span><\/h3>\n<p>The so-called workflows are of particular importance in the ELN component. A \"workflow\" is a generic concept that describes a well-defined sequence of sequential or parallel steps, which are processed as automatically as possible. This can include the execution of an analysis tool or the control and data retrieval of an experimental device. To accommodate such heterogeneity, concrete steps must be implemented as flexibly as possible, since they are highly user- and application-specific. In Figure 1, the types of tools shown in the second layer are used as part of the workflows, so as to implement the actual functionality of the various steps. These can be roughly divided into analysis, visualization, transformation, and transportation tasks. In order to keep the application of these tools as generic as possible, a combination of provided and user-defined tools is accessed. From a user\u2019s perspective, it must be possible to provide such tools in an easy manner, while the execution of each tool must take place in a secure and functional environment. This is especially true for existing tools\u2014e.g., a simple MATLAB<sup id=\"rdp-ebb-cite_ref-MATLAB_30-0\" class=\"reference\"><a href=\"#cite_note-MATLAB-30\">[30]<\/a><\/sup> script\u2014which require certain dependencies to be executed and must be equipped with a suitable interface to be used within a workflow. Depending on their functionality, the tools must in turn access various technical infrastructures. In addition to the use of the repository and computing infrastructure, direct access to devices is also important for more complex data analyses. The automation of a typical workflow of experimenters is only fully possible if data and metadata, created by devices, can be captured. However, such an integration is not trivial due to a heterogeneous device landscape, as well as proprietary data formats and interfaces.<sup id=\"rdp-ebb-cite_ref-HawkerLab07_31-0\" class=\"reference\"><a href=\"#cite_note-HawkerLab07-31\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PotthoffProc19_32-0\" class=\"reference\"><a href=\"#cite_note-PotthoffProc19-32\">[32]<\/a><\/sup> In Kadi4Mat, it should also be possible to use individual tools separately, where appropriate, i.e., outside a workflow. For example, when using the web-based interface, a visualization tool for a custom data format may be used to generate a preview of a datum that can be directly displayed in a web browser.\n<\/p><p>In Figure 2, the current concept for the integration of workflows in Kadi4Mat is shown. Different steps of a workflow can be defined with a graphical node editor. Either a web-based or a desktop-based version of such an editor can be used, the latter running as an ordinary application on a local workstation. With the help of such an editor, the different steps or tools to be executed are defined, linked, and, most importantly, parameterized. The execution of a workflow can be started via an external component called \"Process Manager.\" This component in turn manages several process engines, which take care of executing the workflows. The process engines potentially differ in their implementation and functionality. A simple process engine, for example, could be limited to a sequential execution order of the different tasks, while another one could execute independent tasks in parallel. All engines process the required steps based on the <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> stored in the workflow. With appropriate transport tools, the data and metadata required for each step, as well as the resulting output, can be exported or imported from Kadi4Mat using the existing interfaces of the research data infrastructure. With similar tools, the use of other external data sources becomes possible, and with it the possibility to handle large amounts of data via suitable exchange protocols. The use of locally stored data is also possible when running a workflow on a local workstation.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Brandt_DataSciJourn21_20-1.png\" class=\"image wiki-link\" data-key=\"b4758b76ef8616173357c1178e169633\"><img alt=\"Fig2 Brandt DataSciJourn21 20-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/58\/Fig2_Brandt_DataSciJourn21_20-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Conceptual overview of the workflow architecture. Each workflow is defined using a graphical editor that is either directly integrated into the web-based interface of Kadi4Mat or locally, with a desktop application. The process manager provides an interface for executing workflows and communicates on behalf of the user with multiple process engines, to which the actual execution of workflows is delegated. The engines are responsible for the actual processing of the different steps, based on the information defined in a workflow. Data and metadata can either be stored externally or locally.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Since the reproducibility of the performed steps is a key objective of the workflows, all meaningful information and metadata can be logged along the way. The logging needs to be flexible in order to accommodate different individual or organizational needs, and as such, it is also part of the workflow itself. Workflows can also be shared with other users, for example, via Kadi4Mat. Manual steps may require interaction during the execution of a workflow, for which the system must prompt the user. In summary, the focus of the ELN component thus points in a different direction than in classic ELNs, with the emphasis on the automation of the steps performed. This aspect in particular is similar to systems such as Galaxy<sup id=\"rdp-ebb-cite_ref-AfganTheGal18_17-1\" class=\"reference\"><a href=\"#cite_note-AfganTheGal18-17\">[17]<\/a><\/sup>, which focuses on computational biology, or Taverna<sup id=\"rdp-ebb-cite_ref-WolstencroftTheTav13_33-0\" class=\"reference\"><a href=\"#cite_note-WolstencroftTheTav13-33\">[33]<\/a><\/sup>, a dedicated workflow management system. Nevertheless, some typical features of classic ELNs are also considered in the ELN component, such as the inclusion of handwritten notes.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Repository\">Repository<\/span><\/h3>\n<p>In the repository component, data management is regarded as the central element, especially the structured data storage and exchange. An important aspect is the enrichment of data with corresponding descriptive metadata, which is required for its description, analysis, or search. Many repositories, especially those focused on publishing research data, use the metadata schema provided by DataCite<sup id=\"rdp-ebb-cite_ref-DataCite43_34-0\" class=\"reference\"><a href=\"#cite_note-DataCite43-34\">[34]<\/a><\/sup> and are either directly or heavily based on it. This schema is widely supported and enables the direct publication of data via the corresponding DataCite service. For use cases that go beyond data publications, it is limited in its descriptive power, at the same time. There are comparatively few subject-specific schemas available for engineering and material sciences. Two examples are EngMeta<sup id=\"rdp-ebb-cite_ref-SchemberaEngMeta20_35-0\" class=\"reference\"><a href=\"#cite_note-SchemberaEngMeta20-35\">[35]<\/a><\/sup> and NOMAD Meta Info.<sup id=\"rdp-ebb-cite_ref-GhiringhelliTowards17_36-0\" class=\"reference\"><a href=\"#cite_note-GhiringhelliTowards17-36\">[36]<\/a><\/sup> The first schema is created <i>a priori<\/i> and aims to provide a generic description of computer-aided engineering data, while the second schema is created <i>a posteriori<\/i>, using existing computing inputs and outputs from the database of the NOMAD repository.\n<\/p><p>The second approach is also pursued in a similar way in Kadi4Mat. Instead of a fixed metadata schema, the concrete structure is largely determined by the users themselves, and thus is oriented towards their specific needs. To aid with establishing common metadata vocabularies, a mechanism to create templates is provided. Templates can impose certain restrictions and validations on certain metadata. They are user-defined and can be shared within workgroups or projects, facilitating the establishment of metadata standards. Nevertheless, individual, generic metadata fields, such as a title or description of a data set, can be static. For different use cases such as data analysis, publishing, or the interoperability with other systems, additional conversions must be provided. This is not only necessary because of differing data formats, but also to map vocabularies of different schemas accordingly. Such converted metadata can either represent a subset of existing schemas or require additional fields, such as a license for the re-use of published data. In the long run, the objective in Kadi4Mat is to offer well-defined structures and semantics by making use of ontologies. In the field of materials science, there are ongoing developments in this respect, such as the European Materials Modelling Ontology.<sup id=\"rdp-ebb-cite_ref-GH_EMMO_37-0\" class=\"reference\"><a href=\"#cite_note-GH_EMMO-37\">[37]<\/a><\/sup> However, a bottom-up procedure is considered as a more flexible solution, with the objective to generate an <a href=\"https:\/\/www.limswiki.org\/index.php\/Ontology_(information_science)\" title=\"Ontology (information science)\" class=\"wiki-link\" data-key=\"52d0664bde4b458e81fbc128b911a4a6\">ontology<\/a> from existing metadata and relationships between different data sets. Such a two-pronged approach aims to be functional in the short term, while still staying extensible in the long term<sup id=\"rdp-ebb-cite_ref-GreenbergAMeta09_38-0\" class=\"reference\"><a href=\"#cite_note-GreenbergAMeta09-38\">[38]<\/a><\/sup>, although it heavily depends on how users manage their data and metadata with the options available.\n<\/p><p>In addition to the metadata, the actual data must be managed as well. Here, one can distinguish between data managed directly by Kadi4Mat and linked data. In the simplest form, the former resides on a file system accessible by the repository, which means full control over the data. This requires a copy of each datum to be made available in Kadi4Mat, which makes it less suitable for very large amounts of data. The same applies to data analyses that are to be carried out on external computing infrastructures and must access the data for this purpose. Linked data, on the other hand, can be located on external data storage devices, e.g., high-performance computing infrastructures. This also makes it possible to integrate existing infrastructures and repositories. In these cases, Kadi4Mat can simply offer a view on top of such infrastructures or a more direct integration, depending on the concrete system in question.\n<\/p><p>A further point to be addressed within the repository is the publication of data and metadata, including templates and workflows, that require persistent identifiers to be referenceable. Many existing repositories and systems are already specialized in exactly this use case and offer infrastructures for the long-term archiving of large amounts of data. Thus, an integration of suitable external systems is to be considered for this task in particular. From Kadi4Mat\u2019s point of view, only certain basic requirements have to be ensured in order to enable the publishing of data. These include the assignment of a unique identifier within the system, the provision of metadata and licenses, necessary for a publication, and a basic form of user-guided <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a>. The repository component thus also goes in a different direction than classic repositories. In a typical scientific workflow, it is primarily focused on all steps that take place between the initial data acquisition and the publishing of data. The component is therefore best described as a community repository that manages warm data (i.e., unpublished data that needs further analysis) and enables data exchange within specific communities, e.g., within a research group or project.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<p>Kadi4Mat is built as a web-based application that employs a classic client-server architecture. A graphical front end is provided to be used with a normal web browser as a client, while the server is responsible for handling the back end and the integration of external systems. A high-level overview of the implementation is shown in Figure 3. The front end is based on the classic web technologies JavaScript, HTML, and CSS. In particular, the client-side JavaScript web framework Vue.js<sup id=\"rdp-ebb-cite_ref-VUE_39-0\" class=\"reference\"><a href=\"#cite_note-VUE-39\">[39]<\/a><\/sup> is used. The framework is especially suitable for the creation of single-page web applications (SPA), but it can also be used for individual sections of more classic applications, to incrementally add complex and dynamic user interface components to certain pages. Vue.js is mainly used for the latter, the benefit being a clear separation between the data and the presentation layer, as well as the easier re-use of user interface components. This aspect is combined with server-side rendering. Due to the technologies and standards employed, the use of the front end is currently limited to recent versions of modern web browsers such as Firefox, Chrome, or Edge.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Brandt_DataSciJourn21_20-1.png\" class=\"image wiki-link\" data-key=\"90081b2364dfa0458df68a1fd6604b60\"><img alt=\"Fig3 Brandt DataSciJourn21 20-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0b\/Fig3_Brandt_DataSciJourn21_20-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Overview of the implementation of Kadi4Mat, separated into front end and back end. The front end uses classic web technologies and is usually operated via a web browser. In the back end, the functionality is split into the web and the core component. The former takes care of the external interfaces, while the latter contains most of the core functionality and handles the interfaces of other systems. A plugin component is also shown, which can be used to customize or extend the functionality of the system.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In the back end, the framework Flask<sup id=\"rdp-ebb-cite_ref-Flask_40-0\" class=\"reference\"><a href=\"#cite_note-Flask-40\">[40]<\/a><\/sup> is used for the web component. The framework is implemented in Python and is compatible with the common web server gateway interface (WSGI), which specifies an interface between web servers and Python applications. As a so-called microframework, the functionality of Flask itself is limited to the basic features. This means that most of the functionality, which is unrelated to the web component, has to be added by custom code or suitable libraries. At the same time, more freedom is offered in the concrete choice of technologies. This is in direct contrast to web frameworks such as Django<sup id=\"rdp-ebb-cite_ref-Django_41-0\" class=\"reference\"><a href=\"#cite_note-Django-41\">[41]<\/a><\/sup>, which already provides a lot of functionality from scratch. The web component itself is responsible for handling client requests for specific endpoints and assigning them to the appropriate Python functions. Currently, either HTML or JSON is returned, depending on the endpoint. The latter is used as part of an HTTP <a href=\"https:\/\/www.limswiki.org\/index.php\/Application_programming_interface\" title=\"Application programming interface\" class=\"wiki-link\" data-key=\"36fc319869eba4613cb0854b421b0934\">application programming interface<\/a> (API), to enable an internal and external programmatic data exchange. This API is based on the representational state transfer (REST) paradigm.<sup id=\"rdp-ebb-cite_ref-FieldingArch00_42-0\" class=\"reference\"><a href=\"#cite_note-FieldingArch00-42\">[42]<\/a><\/sup> Support for other exchange formats could also be relevant in the future, particularly for implementing certain exchange formats for interoperability, such as OAI-PMH.<sup id=\"rdp-ebb-cite_ref-OAIProt_43-0\" class=\"reference\"><a href=\"#cite_note-OAIProt-43\">[43]<\/a><\/sup> Especially for handling larger amounts of data, other exchange protocols besides HTTP are considered.\n<\/p><p>A large part of the application consists of the core functionality, which is divided into different modules, as shown in Figure 3. This structure is mainly of an organizational nature. A microservice architecture is currently not implemented. Modules that access external components are particularly noteworthy, which is an aspect that will also be increasingly important in the future. External components can either run on the same hardware as Kadi4Mat itself or on separate systems available via a network interface. For the storage of metadata, the persistence module makes use of the relational database management system <a href=\"https:\/\/www.limswiki.org\/index.php\/PostgreSQL\" title=\"PostgreSQL\" class=\"wiki-link\" data-key=\"a5dd945cdcb63e2d8f7a5edb3a896d82\">PostgreSQL<\/a><sup id=\"rdp-ebb-cite_ref-PostegreSQL_44-0\" class=\"reference\"><a href=\"#cite_note-PostegreSQL-44\">[44]<\/a><\/sup>, while the regular file system stores the actual data. Additionally, the software Elasticsearch<sup id=\"rdp-ebb-cite_ref-Elasticsearch_45-0\" class=\"reference\"><a href=\"#cite_note-Elasticsearch-45\">[45]<\/a><\/sup> is used to index all the metadata that needs to be efficiently searchable. The aforementioned process manager<sup id=\"rdp-ebb-cite_ref-ZschummeIAM21_46-0\" class=\"reference\"><a href=\"#cite_note-ZschummeIAM21-46\">[46]<\/a><\/sup>, which is currently implemented as a command line application, manages the execution of workflows by delegating each execution task to an available process engine.<sup id=\"rdp-ebb-cite_ref-ZschummeIAMEng21_47-0\" class=\"reference\"><a href=\"#cite_note-ZschummeIAMEng21-47\">[47]<\/a><\/sup> While the current implementation of the process engine primarily uses the local file system of the machine on which it is running, users can add steps to synchronize data with the repository to their workflow at will. To increase performance with multiple parallel requests for workflow execution, the requests can be distributed to process engines running on additional servers. By wrapping the process manager with a simple HTTP API, for example, its interface can easily be used over a network. A message broker is used to decouple longer running or periodically executed background tasks from the rest of the application, by delegating them to one or more background worker processes. Apart from using locally managed user accounts or an LDAP system for authentication, Shibboleth<sup id=\"rdp-ebb-cite_ref-WalkerShibb17_48-0\" class=\"reference\"><a href=\"#cite_note-WalkerShibb17-48\">[48]<\/a><\/sup> can be used as well. From a technical point of view, Shibboleth is not a single system, but the interaction of several components, which together enable a distributed authentication procedure. Depending on the type of authentication, user attributes or group affiliations can also be used for authorization purposes in the future.\n<\/p><p>Another component shown in Figure 3 involves the plugins. These can be used to customize or extend the basic functionality of certain procedures or actions, without having to modify or know the corresponding implementation in detail. Unlike the tools in a workflow, plugins make use of predefined hooks to add their custom functionality. While such a plugin has to be installed centrally by the system administrator for all users of a Kadi4Mat instance, the possibilities are also evaluated to be able to make use of individual plugins on the user level.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>The current functionalities of Kadi4Mat can either be utilized via the graphical user interface, with a browser, or via the HTTP API, with a suitable client. On top of the API, a Python library is developed, which makes it especially easy to interact with the different functionalities.<sup id=\"rdp-ebb-cite_ref-SchoofIAM20_49-0\" class=\"reference\"><a href=\"#cite_note-SchoofIAM20-49\">[49]<\/a><\/sup> Besides using the library in Python code, it offers a command-line interface, enabling the integration with other programming or scripting languages.\n<\/p><p>In the following, the most important features of Kadi4Mat are explained, based on its graphical user interface. The focus of the features implemented so far is on the repository component, the topics of structured data management and data exchange in particular, as well as on the workflows, which are a central part of the ELN\u2019s functionality. After logging in to Kadi4Mat, it is possible to create different types of resources. The most important type of resource are the so-called records, which can link arbitrary data with descriptive metadata and serve as basic components that can be used in workflows and future data publications. In principle, a record can be used for all kinds of data, including data from simulations or experiments, and it can be linked to other records of related data sets, e.g., to the descriptions of the software and hardware devices used. \n<\/p><p>The metadata of a record includes both basic metadata, such as title or description, and domain-specific metadata, which can be specified generically, in the form of key\/value pairs. The latter can be defined using a special editor, as shown in Figure 4. With the help of such metadata, a description of subject- and application-specific records becomes possible. This is particularly relevant in an interdisciplinary research field such as materials science, where using a fixed schema would be impracticable, due to the heterogeneity of the data formats and the corresponding metadata. The value of each metadata entry can be of different types, such as simple character strings or numeric types like integers and floating point numbers. Numeric values can also be provided with an arbitrary unit. Furthermore, nested types can be used to represent metadata structures of almost any complexity, for example in the form of lists. The input of such structures can be simplified by templates, which are specified in advance and can be combined as desired. While templates currently offer the same possibilities as the actual metadata, it is planned to add further validation functionalities, such as the specification of a selection of valid values for certain metadata keys. Wherever possible, automatically recorded metadata is also available in each record, such as the creator of the record or the creation date. The actual data of the record can be uploaded by the users and are currently stored on a file system, accessible by Kadi4Mat. It is possible to upload any number of files for each record. This can be helpful when dealing with a series of several hundred images of a simulated microstructure, for example, which all share the same metadata.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Brandt_DataSciJourn21_20-1.png\" class=\"image wiki-link\" data-key=\"7a96dac5d3520a3d46b2965d79c218ae\"><img alt=\"Fig4 Brandt DataSciJourn21 20-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/c3\/Fig4_Brandt_DataSciJourn21_20-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Screenshot of the generic metadata editor, showing the different types of metadata entries currently possible. The last two examples of type dictionary and list contain nested metadata entries. In the upper right corner, a menu is displayed that allows performing various actions, one of which switches to a tree-based overview of the metadata. The ability to select metadata templates is shown in the lower right corner.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The created record can be viewed on an overview page that displays all metadata and linked files. Some common file formats include a preview that is directly integrated into the web browser, such as image files, PDFs, archives, or textual data. Furthermore, the access rights of the record are displayed on its overview page. Currently, two levels of visibility can be set when creating a record: public and private visibility. While public records can be viewed by every logged-in user\u2014i.e., read rights are granted implicitly to each user\u2014private records can initially only be viewed by their creator. Only the creator of a record can perform further actions, such as editing the metadata or deleting the entire record. Figure 5 shows the overview page of a record, including its metadata and the menu to perform the previously mentioned actions. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Brandt_DataSciJourn21_20-1.png\" class=\"image wiki-link\" data-key=\"d40380c6eaa2157c4a1b7026ef4b7173\"><img alt=\"Fig5 Brandt DataSciJourn21 20-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a6\/Fig5_Brandt_DataSciJourn21_20-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Screenshot of a record overview page. The basic metadata is shown, followed by the generic metadata entries (shown as extra metadata). The menu on the top allows various actions to be performed on the current record. The tabs below the menu are used to switch to other views that display the files and other resources associated with the current record, as well as access permissions and a history of metadata revisions.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In order to grant different access rights to other users, even within private records, different roles can be defined for any user in a separate view. Currently, the roles are static, which means that they can be selected from a predefined list and are each linked to the corresponding fine-grained permissions. Because of these permissions, the possibility of custom roles or certain actions being linked to different user attributes becomes possible. In addition to roles for individual users, roles can also be defined for groups. These are simple groupings of several users which, similar to records, can be created and managed by the users themselves. The same roles that can be defined for individual users can be assigned to groups as well. Each member of the group is granted the corresponding access rights automatically. \n<\/p><p>Finally, the overview page of a record also shows the resources linked to it. This refers in particular to the so-called collections. Collections represent simple, logical groupings of multiple records and can thus contribute to a better organization of resources. In terms of an ontology, collections can be regarded as classes, while records inside a collection represent concrete instances of such a class. Like records and groups, collections can be created and managed by users. Records can also be linked to other records. Each record link represents a separate resource, which in turn can contain certain metadata. The ability to specify generic metadata and such resource links already enables a basic ontology-like structure. This structure can be further improved in the future, e.g., by using different types of links, with varying semantics, and by allowing collections to be nested.\n<\/p><p>To be able to find different resources efficiently, especially records, a search function is included in Kadi4Mat. This allows searching in the basic metadata of resources and in the generic metadata of records via keywords or full text search. The values of nested generic metadata entries are flattened before they are indexed in ElasticSearch. This way, a common search mapping can be defined for all kinds of generic metadata. The search results can be sorted and filtered in various ways, for example, by using different user-defined tags or data formats, in the case of records. Figure 6 shows an example search of records, with the corresponding results.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Brandt_DataSciJourn21_20-1.png\" class=\"image wiki-link\" data-key=\"bb844f1d73a0bb3e7d8db3f31c293b5a\"><img alt=\"Fig6 Brandt DataSciJourn21 20-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/6f\/Fig6_Brandt_DataSciJourn21_20-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> Screenshot of the search functionality of records with the corresponding search results. In addition to providing a simple query for searching the basic metadata of a record, the generic metadata can also be searched by specifying desired keys, types, or values. The searchable types are derived from the actual types of the generic metadata entries, e.g., integers and floating point numbers are grouped together as numeric type. Various other options are offered for filtering and sorting the search results.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>While the execution of workflows, via the web interfaces, and the ability to add user-defined tools are still under development, it is possible to define a workflow using a graphical node editor, running in the web browser. Figure 7 shows a simple example workflow created with this editor. A selection of predefined nodes can be combined and parameterized, while the resulting workflow can be downloaded. A custom JSON-based format is currently used to store the representation of a workflow. This format contains all the information for the node editor to correctly display the workflow and to derive a functional workflow representation for execution. The downloaded workflow file can be executed directly on a local workstation by using the command line interface of the process manager. All tools to actually run such a workflow need to be installed beforehand. A selection of tools is provided for various tasks<sup id=\"rdp-ebb-cite_ref-ZschummeIAMNodes21_50-0\" class=\"reference\"><a href=\"#cite_note-ZschummeIAMNodes21-50\">[50]<\/a><\/sup>, including connecting to a Kadi4Mat instance by using a suitable wrapper on top of the aforementioned API library. Several common use cases have already been implemented, including the task of extracting metadata from Excel spreadsheets, often used to replace an actual ELN system, and importing it into Kadi4Mat. An overview of such a workflow is shown in Figure 8. Developments are also underway for data science applications, especially in the field of machine learning. The combination of the ELN and the repository fits particularly well with the requirements of such applications, which typically require lots of high-quality input data to function well.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Brandt_DataSciJourn21_20-1.png\" class=\"image wiki-link\" data-key=\"f81a7fb335b62f5e615e0dfc930ca8f8\"><img alt=\"Fig7 Brandt DataSciJourn21 20-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/49\/Fig7_Brandt_DataSciJourn21_20-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 7.<\/b> Screenshot of a workflow created with the web-based node editor. Several String input nodes are shown, as well as a special node that prompts the user to enter a file (UserInput: File). The two tools <i>mkdir<\/i> and <i>ImageJMacro<\/i> are used to create a new directory and to execute an ImageJ<sup id=\"rdp-ebb-cite_ref-SchindelinTheImage15_51-0\" class=\"reference\"><a href=\"#cite_note-SchindelinTheImage15-51\">[51]<\/a><\/sup> macro file, respectively. The latter uses the input file the user was asked for. Except for the input nodes, all nodes are connected via an explicit dependency.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Brandt_DataSciJourn21_20-1.png\" class=\"image wiki-link\" data-key=\"d7c97cef65d783e58ef20c11f0ea27fe\"><img alt=\"Fig8 Brandt DataSciJourn21 20-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/72\/Fig8_Brandt_DataSciJourn21_20-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 8.<\/b> Overview of an exemplary workflow using Kadi4Mat. The starting point is raw data and corresponding metadata stored in an Excel spreadsheet. The tools used in this workflow are divided into tools for data handling and tools for data transport, the latter referring to the Kadi4Mat integration. In a first conversion step, the metadata are transformed into a format readable by the API of Kadi4Mat and linked to the raw data by creating a new record. The raw data is further analyzed using the metadata stored in Kadi4Mat. Finally, the result of the analysis is plotted and both data sets are uploaded to Kadi4Mat as records. All records can be linked to each other in a further step, either as part of the workflow or separately.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The development and current functionality of the research data infrastructure Kadi4Mat is presented. The objective of this infrastructure is to combine the features of an ELN and a repository in such a way that researchers can be supported throughout the whole research process. The ongoing development aims at covering the heterogeneous use cases of materials science disciplines. For this purpose, flexible metadata schemas, workflows, and tools are especially important, as is the use of custom installations and instances. The basic functionality of the repository component is largely given by the features already implemented and can be used with a graphical as well as a programmatic interface. This includes, above all, uploading, managing, and exchanging data, as well as the associated metadata. The latter can be defined with a flexible metadata editor to accommodate the needs of different users and workgroups. A search functionality enables the efficient retrieval of the data. The essential infrastructure for workflows is implemented as a central part of the ELN component. Simple workflows can be defined with an initial version of the web-based node editor and executed locally using provided tools and the process manager\u2019s command-line interface. Both main components are improved continuously. \n<\/p><p>Various other features that have not yet been mentioned as part of the concept are planned or are already in the conception stage. These include the optional connection of several Kadi4Mat instances, a more direct, low-level access to data, and the integration of an app store, for the central administration of tools and plugins.\n<\/p><p>The development of Kadi4Mat largely follows a bottom-up approach. Instead of developing concepts in advance, to cover as many use cases as possible, a basic technical infrastructure is established first. On this basis, further steps are evaluated in exchange with interested users and by implementing best practice examples. Due to the heterogeneous nature of materials science, most features are kept very generic. The concrete structuring of the data storage, the metadata, and the workflows is largely left to the users. As a positive side effect, an extension of the research data infrastructure to other disciplines is possible in the future.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This work is supported by the Federal Ministry of Education and Research (BMBF) in the projects FestBatt (project number 03XP0174E) and as part of the Excellence Strategy of the German Federal and State Governments, by the German Research Foundation (DFG) in the projects POLiS (project number 390874152) and SuLMaSS (project number 391128822) and by the Ministry of Science, Research and Art Baden-W\u00fcrttemberg in the project MoMaF \u2013 Science Data Center, with funds from the state digitization strategy digital@bw (project number 57). The authors are also grateful for the editorial support of Leon Geisen.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors have no competing interests to declare.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-SandfeldStrateg18-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SandfeldStrateg18_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Sandfeld, S.; Dahmen, T.; Fischer, F.O.R. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.tib.eu\/en\/search\/id\/TIBKAT%3A1028913559\/\" target=\"_blank\">\"Strategiepapier - Digitale Transformation in der Materialwissenschaft und Werkstofftechnik\"<\/a>. Deutsche Gesellschaft f\u00fcr Materialkunde e.V<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.tib.eu\/en\/search\/id\/TIBKAT%3A1028913559\/\" target=\"_blank\">https:\/\/www.tib.eu\/en\/search\/id\/TIBKAT%3A1028913559\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Strategiepapier+-+Digitale+Transformation+in+der+Materialwissenschaft+und+Werkstofftechnik&rft.atitle=&rft.aulast=Sandfeld%2C+S.%3B+Dahmen%2C+T.%3B+Fischer%2C+F.O.R.+et+al.&rft.au=Sandfeld%2C+S.%3B+Dahmen%2C+T.%3B+Fischer%2C+F.O.R.+et+al.&rft.date=2018&rft.pub=Deutsche+Gesellschaft+f%C3%BCr+Materialkunde+e.V&rft_id=https%3A%2F%2Fwww.tib.eu%2Fen%2Fsearch%2Fid%2FTIBKAT%253A1028913559%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeyTheData03-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeyTheData03_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Hey, T.; Trefethen, A. (2003). \"Chapter 36: The Data Deluge: An e\u2010Science Perspective\". In Berman, F.; Fox, G.; Hey, T.. <i>Grid Computing: Making the Global Infrastructure a Reality<\/i>. John Wiley & Sons, Ltd. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2F0470867167.ch36\" target=\"_blank\">10.1002\/0470867167.ch36<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780470867167.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+36%3A+The+Data+Deluge%3A+An+e%E2%80%90Science+Perspective&rft.atitle=Grid+Computing%3A+Making+the+Global+Infrastructure+a+Reality&rft.aulast=Hey%2C+T.%3B+Trefethen%2C+A.&rft.au=Hey%2C+T.%3B+Trefethen%2C+A.&rft.date=2003&rft.pub=John+Wiley+%26+Sons%2C+Ltd&rft_id=info:doi\/10.1002%2F0470867167.ch36&rft.isbn=9780470867167&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeyTheFourth09-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeyTheFourth09_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Hey, T.; Tansley, S.; Tolle, K. (2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/fourth-paradigm-data-intensive-scientific-discovery\/\" target=\"_blank\"><i>The Fourth Paradigm: Data-Intensive Scientific Discovery<\/i><\/a>. Microsoft Research. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780982544204<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/fourth-paradigm-data-intensive-scientific-discovery\/\" target=\"_blank\">https:\/\/www.microsoft.com\/en-us\/research\/publication\/fourth-paradigm-data-intensive-scientific-discovery\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Fourth+Paradigm%3A+Data-Intensive+Scientific+Discovery&rft.aulast=Hey%2C+T.%3B+Tansley%2C+S.%3B+Tolle%2C+K.&rft.au=Hey%2C+T.%3B+Tansley%2C+S.%3B+Tolle%2C+K.&rft.date=2009&rft.pub=Microsoft+Research&rft.isbn=9780982544204&rft_id=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fresearch%2Fpublication%2Ffourth-paradigm-data-intensive-scientific-discovery%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HillMaterials16-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HillMaterials16_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-HillMaterials16_4-1\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hill, J.; Mulholland, G.; Persson, K. et al. (2016). \"Materials science with large-scale data and informatics: Unlocking new opportunities\". <i>MRS Bulletin<\/i> <b>41<\/b> (5): 399\u2013409. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1557%2Fmrs.2016.93\" target=\"_blank\">10.1557\/mrs.2016.93<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Materials+science+with+large-scale+data+and+informatics%3A+Unlocking+new+opportunities&rft.jtitle=MRS+Bulletin&rft.aulast=Hill%2C+J.%3B+Mulholland%2C+G.%3B+Persson%2C+K.+et+al.&rft.au=Hill%2C+J.%3B+Mulholland%2C+G.%3B+Persson%2C+K.+et+al.&rft.date=2016&rft.volume=41&rft.issue=5&rft.pages=399%E2%80%93409&rft_id=info:doi\/10.1557%2Fmrs.2016.93&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NaughtonMaking16-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NaughtonMaking16_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Naughton, L.; Kernohan, D. (2016). \"Making sense of journal research data policies\". <i>Insights<\/i> <b>29<\/b> (1): 84\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1629%2Fuksg.284\" target=\"_blank\">10.1629\/uksg.284<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Making+sense+of+journal+research+data+policies&rft.jtitle=Insights&rft.aulast=Naughton%2C+L.%3B+Kernohan%2C+D.&rft.au=Naughton%2C+L.%3B+Kernohan%2C+D.&rft.date=2016&rft.volume=29&rft.issue=1&rft.pages=84%E2%80%939&rft_id=info:doi\/10.1629%2Fuksg.284&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PampelMaking13-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PampelMaking13_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pampel, H.; Vierkant, P.; Scholze, F. et al. (2013). \"Making Research Data Repositories Visible: The re3data.org Registry\". <i>PLoS One<\/i> <b>8<\/b> (11): e78080. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0078080\" target=\"_blank\">10.1371\/journal.pone.0078080<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Making+Research+Data+Repositories+Visible%3A+The+re3data.org+Registry&rft.jtitle=PLoS+One&rft.aulast=Pampel%2C+H.%3B+Vierkant%2C+P.%3B+Scholze%2C+F.+et+al.&rft.au=Pampel%2C+H.%3B+Vierkant%2C+P.%3B+Scholze%2C+F.+et+al.&rft.date=2013&rft.volume=8&rft.issue=11&rft.pages=e78080&rft_id=info:doi\/10.1371%2Fjournal.pone.0078080&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SansoneFAIR19-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SansoneFAIR19_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sansone, S.-A.; McQuilton, P.; Rocca-Serra, P. et al. (2019). \"FAIRsharing as a community approach to standards, repositories and policies\". <i>Nature Biotechnology<\/i> <b>37<\/b>: 358\u201367. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41587-019-0080-8\" target=\"_blank\">10.1038\/s41587-019-0080-8<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=FAIRsharing+as+a+community+approach+to+standards%2C+repositories+and+policies&rft.jtitle=Nature+Biotechnology&rft.aulast=Sansone%2C+S.-A.%3B+McQuilton%2C+P.%3B+Rocca-Serra%2C+P.+et+al.&rft.au=Sansone%2C+S.-A.%3B+McQuilton%2C+P.%3B+Rocca-Serra%2C+P.+et+al.&rft.date=2019&rft.volume=37&rft.pages=358%E2%80%9367&rft_id=info:doi\/10.1038%2Fs41587-019-0080-8&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JainComment13-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JainComment13_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jain, A.; Ong, S.P.; Hautier, G. et al. (2013). \"Commentary: The Materials Project: A materials genome approach to accelerating materials innovation\". <i>APL Materials<\/i> <b>1<\/b> (1): 011002. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1063%2F1.4812323\" target=\"_blank\">10.1063\/1.4812323<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Commentary%3A+The+Materials+Project%3A+A+materials+genome+approach+to+accelerating+materials+innovation&rft.jtitle=APL+Materials&rft.aulast=Jain%2C+A.%3B+Ong%2C+S.P.%3B+Hautier%2C+G.+et+al.&rft.au=Jain%2C+A.%3B+Ong%2C+S.P.%3B+Hautier%2C+G.+et+al.&rft.date=2013&rft.volume=1&rft.issue=1&rft.pages=011002&rft_id=info:doi\/10.1063%2F1.4812323&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DraxlNOMAD18-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DraxlNOMAD18_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Draxl, C.; Scheffler, M. (2018). \"NOMAD: The FAIR concept for big data-driven materials science\". <i>MRS Bulletin<\/i> <b>43<\/b> (9): 676\u201382. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1557%2Fmrs.2018.208\" target=\"_blank\">10.1557\/mrs.2018.208<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=NOMAD%3A+The+FAIR+concept+for+big+data-driven+materials+science&rft.jtitle=MRS+Bulletin&rft.aulast=Draxl%2C+C.%3B+Scheffler%2C+M.&rft.au=Draxl%2C+C.%3B+Scheffler%2C+M.&rft.date=2018&rft.volume=43&rft.issue=9&rft.pages=676%E2%80%9382&rft_id=info:doi\/10.1557%2Fmrs.2018.208&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CKANHome-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CKANHome_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ckan.org\/\" target=\"_blank\">\"CKAN\"<\/a>. CKAN Association<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ckan.org\/\" target=\"_blank\">https:\/\/ckan.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 May 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CKAN&rft.atitle=&rft.pub=CKAN+Association&rft_id=https%3A%2F%2Fckan.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KinganIntro07-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KinganIntro07_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">King, G. (2007). \"An Introduction to the Dataverse Network as an Infrastructure for Data Sharing\". <i>Sociological Methods & Research<\/i> <b>36<\/b> (2): 173\u201399. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F0049124107306660\" target=\"_blank\">10.1177\/0049124107306660<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Introduction+to+the+Dataverse+Network+as+an+Infrastructure+for+Data+Sharing&rft.jtitle=Sociological+Methods+%26+Research&rft.aulast=King%2C+G.&rft.au=King%2C+G.&rft.date=2007&rft.volume=36&rft.issue=2&rft.pages=173%E2%80%9399&rft_id=info:doi\/10.1177%2F0049124107306660&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmithDSpace03-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmithDSpace03_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smith, M.; Barton, M.; Bass, M. et al. (2003). \"DSpace: An Open Source Dynamic Digital Repository\". <i>D-Lib Magazine<\/i> <b>9<\/b> (1). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1045%2Fjanuary2003-smith\" target=\"_blank\">10.1045\/january2003-smith<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DSpace%3A+An+Open+Source+Dynamic+Digital+Repository&rft.jtitle=D-Lib+Magazine&rft.aulast=Smith%2C+M.%3B+Barton%2C+M.%3B+Bass%2C+M.+et+al.&rft.au=Smith%2C+M.%3B+Barton%2C+M.%3B+Bass%2C+M.+et+al.&rft.date=2003&rft.volume=9&rft.issue=1&rft_id=info:doi\/10.1045%2Fjanuary2003-smith&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-InvenioHome-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-InvenioHome_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/invenio-software.org\/\" target=\"_blank\">\"Invenio\"<\/a>. CERN<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/invenio-software.org\/\" target=\"_blank\">https:\/\/invenio-software.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 May 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Invenio&rft.atitle=&rft.pub=CERN&rft_id=https%3A%2F%2Finvenio-software.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZenodoHome-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZenodoHome_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">European Organization for Nuclear Research (2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.zenodo.org\/\" target=\"_blank\">\"Zenodo\"<\/a>. CERN. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.25495%2F7GXK-RD71\" target=\"_blank\">10.25495\/7GXK-RD71<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.zenodo.org\/\" target=\"_blank\">https:\/\/www.zenodo.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Zenodo&rft.atitle=&rft.aulast=European+Organization+for+Nuclear+Research&rft.au=European+Organization+for+Nuclear+Research&rft.date=2013&rft.pub=CERN&rft_id=info:doi\/10.25495%2F7GXK-RD71&rft_id=https%3A%2F%2Fwww.zenodo.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AmorimAComp16-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AmorimAComp16_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Amorim, R.C.; Castro, J.A.; da Silva, J.R. et al. (2017). \"A comparison of research data management platforms: Architecture, flexible metadata and interoperability\". <i>Universal Access in the Information Society<\/i> <b>16<\/b>: 851\u201362. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10209-016-0475-y\" target=\"_blank\">10.1007\/s10209-016-0475-y<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comparison+of+research+data+management+platforms%3A+Architecture%2C+flexible+metadata+and+interoperability&rft.jtitle=Universal+Access+in+the+Information+Society&rft.aulast=Amorim%2C+R.C.%3B+Castro%2C+J.A.%3B+da+Silva%2C+J.R.+et+al.&rft.au=Amorim%2C+R.C.%3B+Castro%2C+J.A.%3B+da+Silva%2C+J.R.+et+al.&rft.date=2017&rft.volume=16&rft.pages=851%E2%80%9362&rft_id=info:doi\/10.1007%2Fs10209-016-0475-y&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RubachaARev11-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RubachaARev11_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rubacha, M.; Rattan, A.K.; Hosselet, S.C. (2011). \"A Review of Electronic Laboratory Notebooks Available in the Market Today\". <i>SLAS Technology<\/i> <b>16<\/b> (1). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jala.2009.01.002\" target=\"_blank\">10.1016\/j.jala.2009.01.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Review+of+Electronic+Laboratory+Notebooks+Available+in+the+Market+Today&rft.jtitle=SLAS+Technology&rft.aulast=Rubacha%2C+M.%3B+Rattan%2C+A.K.%3B+Hosselet%2C+S.C.&rft.au=Rubacha%2C+M.%3B+Rattan%2C+A.K.%3B+Hosselet%2C+S.C.&rft.date=2011&rft.volume=16&rft.issue=1&rft_id=info:doi\/10.1016%2Fj.jala.2009.01.002&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AfganTheGal18-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AfganTheGal18_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-AfganTheGal18_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Afgan, E.; Baker, D.; Batut, B. et al. (2018). \"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2018 update\". <i>Nucleic Acids Research<\/i> <b>46<\/b> (W1). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgky379\" target=\"_blank\">10.1093\/nar\/gky379<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Galaxy+platform+for+accessible%2C+reproducible+and+collaborative+biomedical+analyses%3A+2018+update&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Afgan%2C+E.%3B+Baker%2C+D.%3B+Batut%2C+B.+et+al.&rft.au=Afgan%2C+E.%3B+Baker%2C+D.%3B+Batut%2C+B.+et+al.&rft.date=2018&rft.volume=46&rft.issue=W1&rft_id=info:doi\/10.1093%2Fnar%2Fgky379&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KluyverJupyter16-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KluyverJupyter16_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Kluyver, T.; Ragan-Kelley, B.; P\u00e9rez, F. et al. (2016). \"Jupyter Notebooks\u2014A publishing format for reproducible computational workflows\". In Loizides, F.; Schmidt, B.. <i>Positioning and Power in Academic Publishing: Players, Agents and Agendas<\/i>. IOS Press. pp. 87\u201390. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2F978-1-61499-649-1-87\" target=\"_blank\">10.3233\/978-1-61499-649-1-87<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Jupyter+Notebooks%E2%80%94A+publishing+format+for+reproducible+computational+workflows&rft.atitle=Positioning+and+Power+in+Academic+Publishing%3A+Players%2C+Agents+and+Agendas&rft.aulast=Kluyver%2C+T.%3B+Ragan-Kelley%2C+B.%3B+P%C3%A9rez%2C+F.+et+al.&rft.au=Kluyver%2C+T.%3B+Ragan-Kelley%2C+B.%3B+P%C3%A9rez%2C+F.+et+al.&rft.date=2016&rft.pages=pp.%26nbsp%3B87%E2%80%9390&rft.pub=IOS+Press&rft_id=info:doi\/10.3233%2F978-1-61499-649-1-87&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BirdLab13-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BirdLab13_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bird, C.L.; Willoughby, C.; Frey, J.G. (2013). \"Laboratory notebooks in the digital era: The role of ELNs in record keeping for chemistry and other sciences\". <i>Chemical Society Reviews<\/i> <b>42<\/b> (20): 8157\u20138175. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1039%2FC3CS60122F\" target=\"_blank\">10.1039\/C3CS60122F<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+notebooks+in+the+digital+era%3A+The+role+of+ELNs+in+record+keeping+for+chemistry+and+other+sciences&rft.jtitle=Chemical+Society+Reviews&rft.aulast=Bird%2C+C.L.%3B+Willoughby%2C+C.%3B+Frey%2C+J.G.&rft.au=Bird%2C+C.L.%3B+Willoughby%2C+C.%3B+Frey%2C+J.G.&rft.date=2013&rft.volume=42&rft.issue=20&rft.pages=8157%E2%80%938175&rft_id=info:doi\/10.1039%2FC3CS60122F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElliottThink09-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ElliottThink09_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Elliott, M.H. (2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20110520065023\/http:\/\/www.scientificcomputing.com\/articles-IN-Thinking-Beyond-ELN-120809.aspx\" target=\"_blank\">\"Thinking Beyond ELN\"<\/a>. <i>Scientific Computing<\/i> <b>26<\/b> (6): 6\u201310. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.scientificcomputing.com\/articles-IN-Thinking-Beyond-ELN-120809.aspx\" target=\"_blank\">the original<\/a> on 20 May 2011<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20110520065023\/http:\/\/www.scientificcomputing.com\/articles-IN-Thinking-Beyond-ELN-120809.aspx\" target=\"_blank\">https:\/\/web.archive.org\/web\/20110520065023\/http:\/\/www.scientificcomputing.com\/articles-IN-Thinking-Beyond-ELN-120809.aspx<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Thinking+Beyond+ELN&rft.jtitle=Scientific+Computing&rft.aulast=Elliott%2C+M.H.&rft.au=Elliott%2C+M.H.&rft.date=2009&rft.volume=26&rft.issue=6&rft.pages=6%E2%80%9310&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20110520065023%2Fhttp%3A%2F%2Fwww.scientificcomputing.com%2Farticles-IN-Thinking-Beyond-ELN-120809.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorTheStatus06-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TaylorTheStatus06_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-TaylorTheStatus06_21-1\">21.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, K.T. (2006). \"The status of electronic laboratory notebooks for chemistry and biology\". <i>Current Opinion in Drug Discovery and Development<\/i> <b>9<\/b> (3): 348\u201353. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16729731\" target=\"_blank\">16729731<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+status+of+electronic+laboratory+notebooks+for+chemistry+and+biology&rft.jtitle=Current+Opinion+in+Drug+Discovery+and+Development&rft.aulast=Taylor%2C+K.T.&rft.au=Taylor%2C+K.T.&rft.date=2006&rft.volume=9&rft.issue=3&rft.pages=348%E2%80%9353&rft_id=info:pmid\/16729731&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarpiElabFTW17-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CarpiElabFTW17_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Carpi, N.; Minges, A.; Piel, M. (2017). \"eLabFTW: An open source laboratory notebook for research labs\". <i>Journal of Open Source Software<\/i> <b>2<\/b> (12): 146. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.21105%2Fjoss.00146\" target=\"_blank\">10.21105\/joss.00146<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=eLabFTW%3A+An+open+source+laboratory+notebook+for+research+labs&rft.jtitle=Journal+of+Open+Source+Software&rft.aulast=Carpi%2C+N.%3B+Minges%2C+A.%3B+Piel%2C+M.&rft.au=Carpi%2C+N.%3B+Minges%2C+A.%3B+Piel%2C+M.&rft.date=2017&rft.volume=2&rft.issue=12&rft.pages=146&rft_id=info:doi\/10.21105%2Fjoss.00146&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScinoteHome-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScinoteHome_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.scinote.net\/\" target=\"_blank\">\"SciNote\"<\/a>. SciNote LLC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.scinote.net\/\" target=\"_blank\">https:\/\/www.scinote.net\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 May 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SciNote&rft.atitle=&rft.pub=SciNote+LLC&rft_id=https%3A%2F%2Fwww.scinote.net%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TremouilhacChemotionELN17-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TremouilhacChemotionELN17_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tremouilhac, P.; Nguyen, A.; Huang, Y.-C. et al. (2017). \"Chemotion ELN: An open source electronic lab notebook for chemists in academia\". <i>Journal of Cheminformatics<\/i> <b>9<\/b>: 54. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13321-017-0240-0\" target=\"_blank\">10.1186\/s13321-017-0240-0<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Chemotion+ELN%3A+An+open+source+electronic+lab+notebook+for+chemists+in+academia&rft.jtitle=Journal+of+Cheminformatics&rft.aulast=Tremouilhac%2C+P.%3B+Nguyen%2C+A.%3B+Huang%2C+Y.-C.+et+al.&rft.au=Tremouilhac%2C+P.%3B+Nguyen%2C+A.%3B+Huang%2C+Y.-C.+et+al.&rft.date=2017&rft.volume=9&rft.pages=54&rft_id=info:doi\/10.1186%2Fs13321-017-0240-0&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeidornShed08-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeidornShed08_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Heidorn, P.B. (2008). \"Shedding Light on the Dark Data in the Long Tail of Science\". <i>Library Trends<\/i> <b>57<\/b> (2): 280\u201399. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1353%2Flib.0.0036\" target=\"_blank\">10.1353\/lib.0.0036<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Shedding+Light+on+the+Dark+Data+in+the+Long+Tail+of+Science&rft.jtitle=Library+Trends&rft.aulast=Heidorn%2C+P.B.&rft.au=Heidorn%2C+P.B.&rft.date=2008&rft.volume=57&rft.issue=2&rft.pages=280%E2%80%9399&rft_id=info:doi\/10.1353%2Flib.0.0036&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarusiVirt10-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CarusiVirt10_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Carusi, A.; Reimer, T.F. (17 January 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/hdl.handle.net\/10044\/1\/18568\" target=\"_blank\">\"Virtual Research Environment Collaborative Landscape Study\"<\/a>. JISC. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.25561%2F18568\" target=\"_blank\">10.25561\/18568<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/hdl.handle.net\/10044\/1\/18568\" target=\"_blank\">http:\/\/hdl.handle.net\/10044\/1\/18568<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Virtual+Research+Environment+Collaborative+Landscape+Study&rft.atitle=&rft.aulast=Carusi%2C+A.%3B+Reimer%2C+T.F.&rft.au=Carusi%2C+A.%3B+Reimer%2C+T.F.&rft.date=17+January+2010&rft.pub=JISC&rft_id=info:doi\/10.25561%2F18568&rft_id=http%3A%2F%2Fhdl.handle.net%2F10044%2F1%2F18568&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilkinsonTheFAIR16-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilkinsonTheFAIR16_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175\" target=\"_blank\">\"The FAIR Guiding Principles for scientific data management and stewardship\"<\/a>. <i>Scientific Data<\/i> <b>3<\/b>: 160018. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fsdata.2016.18\" target=\"_blank\">10.1038\/sdata.2016.18<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4792175\/\" target=\"_blank\">PMC4792175<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26978244\" target=\"_blank\">26978244<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4792175<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+FAIR+Guiding+Principles+for+scientific+data+management+and+stewardship&rft.jtitle=Scientific+Data&rft.aulast=Wilkinson%2C+M.D.%3B+Dumontier%2C+M.%3B+Aalbersberg%2C+I.J.+et+al.&rft.au=Wilkinson%2C+M.D.%3B+Dumontier%2C+M.%3B+Aalbersberg%2C+I.J.+et+al.&rft.date=2016&rft.volume=3&rft.pages=160018&rft_id=info:doi\/10.1038%2Fsdata.2016.18&rft_id=info:pmc\/PMC4792175&rft_id=info:pmid\/26978244&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4792175&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Kadi4MatHome-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Kadi4MatHome_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/kadi.iam-cms.kit.edu\/\" target=\"_blank\">\"Kadi4Mat\"<\/a>. Karlsruhe Institute of Technology<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/kadi.iam-cms.kit.edu\/\" target=\"_blank\">https:\/\/kadi.iam-cms.kit.edu\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Kadi4Mat&rft.atitle=&rft.pub=Karlsruhe+Institute+of+Technology&rft_id=https%3A%2F%2Fkadi.iam-cms.kit.edu%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrandtIAM20-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrandtIAM20_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brandt, N.; Griem, L.; Hermann, C. et al. (2020). \"IAM-CMS\/kadi: Kadi4Mat (Version 0.6.0)\". <i>Zenodo<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5281%2Fzenodo.4507826\" target=\"_blank\">10.5281\/zenodo.4507826<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=IAM-CMS%2Fkadi%3A+Kadi4Mat+%28Version+0.6.0%29&rft.jtitle=Zenodo&rft.aulast=Brandt%2C+N.%3B+Griem%2C+L.%3B+Hermann%2C+C.+et+al.&rft.au=Brandt%2C+N.%3B+Griem%2C+L.%3B+Hermann%2C+C.+et+al.&rft.date=2020&rft_id=info:doi\/10.5281%2Fzenodo.4507826&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MATLAB-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MATLAB_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.mathworks.com\/products\/matlab.html\" target=\"_blank\">\"MATLAB\"<\/a>. MathWorks<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.mathworks.com\/products\/matlab.html\" target=\"_blank\">https:\/\/www.mathworks.com\/products\/matlab.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 January 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MATLAB&rft.atitle=&rft.pub=MathWorks&rft_id=https%3A%2F%2Fwww.mathworks.com%2Fproducts%2Fmatlab.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HawkerLab07-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HawkerLab07_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hawker, C.D. (2007). \"Laboratory Automation: Total and Subtotal\". <i>Clinics in Laboratory Medicine<\/i> <b>27<\/b> (4): 749\u201370. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cll.2007.07.010\" target=\"_blank\">10.1016\/j.cll.2007.07.010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+Automation%3A+Total+and+Subtotal&rft.jtitle=Clinics+in+Laboratory+Medicine&rft.aulast=Hawker%2C+C.D.&rft.au=Hawker%2C+C.D.&rft.date=2007&rft.volume=27&rft.issue=4&rft.pages=749%E2%80%9370&rft_id=info:doi\/10.1016%2Fj.cll.2007.07.010&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PotthoffProc19-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PotthoffProc19_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Potthoff, J.; Tremouilhac, P.; Hodapp, P. et al. (2019). \"Procedures for systematic capture and management of analytical data in academia\". <i>Analytica Chimica Acta: X<\/i> <b>1<\/b>: 100007. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.acax.2019.100007\" target=\"_blank\">10.1016\/j.acax.2019.100007<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Procedures+for+systematic+capture+and+management+of+analytical+data+in+academia&rft.jtitle=Analytica+Chimica+Acta%3A+X&rft.aulast=Potthoff%2C+J.%3B+Tremouilhac%2C+P.%3B+Hodapp%2C+P.+et+al.&rft.au=Potthoff%2C+J.%3B+Tremouilhac%2C+P.%3B+Hodapp%2C+P.+et+al.&rft.date=2019&rft.volume=1&rft.pages=100007&rft_id=info:doi\/10.1016%2Fj.acax.2019.100007&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WolstencroftTheTav13-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WolstencroftTheTav13_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wolstencroft, K.; Haines, R.; Fellows, D. et al. (2013). \"The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud\". <i>Nucleic Acids Research<\/i> <b>41<\/b> (W1): W557\u2013W561. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkt328\" target=\"_blank\">10.1093\/nar\/gkt328<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Taverna+workflow+suite%3A+designing+and+executing+workflows+of+Web+Services+on+the+desktop%2C+web+or+in+the+cloud&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Wolstencroft%2C+K.%3B+Haines%2C+R.%3B+Fellows%2C+D.+et+al.&rft.au=Wolstencroft%2C+K.%3B+Haines%2C+R.%3B+Fellows%2C+D.+et+al.&rft.date=2013&rft.volume=41&rft.issue=W1&rft.pages=W557%E2%80%93W561&rft_id=info:doi\/10.1093%2Fnar%2Fgkt328&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DataCite43-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DataCite43_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">DataCite Metadata Working Group (16 August 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/schema.datacite.org\/meta\/kernel-4.3\/\" target=\"_blank\">\"DataCite Metadata Schema for the Publication and Citation of Research Data. Version 4.3\"<\/a>. DataCite e.V<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/schema.datacite.org\/meta\/kernel-4.3\/\" target=\"_blank\">https:\/\/schema.datacite.org\/meta\/kernel-4.3\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=DataCite+Metadata+Schema+for+the+Publication+and+Citation+of+Research+Data.+Version+4.3&rft.atitle=&rft.aulast=DataCite+Metadata+Working+Group&rft.au=DataCite+Metadata+Working+Group&rft.date=16+August+2019&rft.pub=DataCite+e.V&rft_id=https%3A%2F%2Fschema.datacite.org%2Fmeta%2Fkernel-4.3%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchemberaEngMeta20-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchemberaEngMeta20_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schembera, B.; Iglezakis, D. (2020). \"EngMeta: Metadata for computational engineering\". <i>International Journal of Metadata, Semantics and Ontologies<\/i> <b>14<\/b> (1): 26\u201338. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1504%2FIJMSO.2020.107792\" target=\"_blank\">10.1504\/IJMSO.2020.107792<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EngMeta%3A+Metadata+for+computational+engineering&rft.jtitle=International+Journal+of+Metadata%2C+Semantics+and+Ontologies&rft.aulast=Schembera%2C+B.%3B+Iglezakis%2C+D.&rft.au=Schembera%2C+B.%3B+Iglezakis%2C+D.&rft.date=2020&rft.volume=14&rft.issue=1&rft.pages=26%E2%80%9338&rft_id=info:doi\/10.1504%2FIJMSO.2020.107792&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GhiringhelliTowards17-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GhiringhelliTowards17_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ghiringhelli, L.M.; Carbogno, C.; Levchenko, S. et al. (2017). \"Towards efficient data exchange and sharing for big-data driven materials science: metadata and data formats\". <i>Computational Materials<\/i> <b>3<\/b>: 46. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41524-017-0048-5\" target=\"_blank\">10.1038\/s41524-017-0048-5<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+efficient+data+exchange+and+sharing+for+big-data+driven+materials+science%3A+metadata+and+data+formats&rft.jtitle=Computational+Materials&rft.aulast=Ghiringhelli%2C+L.M.%3B+Carbogno%2C+C.%3B+Levchenko%2C+S.+et+al.&rft.au=Ghiringhelli%2C+L.M.%3B+Carbogno%2C+C.%3B+Levchenko%2C+S.+et+al.&rft.date=2017&rft.volume=3&rft.pages=46&rft_id=info:doi\/10.1038%2Fs41524-017-0048-5&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GH_EMMO-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GH_EMMO_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">European Materials Modelling Council. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/emmo-repo\" target=\"_blank\">\"European Materials and Modelling Ontology\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/emmo-repo\" target=\"_blank\">https:\/\/github.com\/emmo-repo<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 24 May 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=European+Materials+and+Modelling+Ontology&rft.atitle=GitHub&rft.aulast=European+Materials+Modelling+Council&rft.au=European+Materials+Modelling+Council&rft_id=https%3A%2F%2Fgithub.com%2Femmo-repo&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GreenbergAMeta09-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GreenbergAMeta09_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Greenberg, J.; White, H.C.; Carrier, S. et al. (2009). \"A Metadata Best Practice for a Scientific Data Repository\". <i>Journal of Library Metadata<\/i> <b>9<\/b> (3\u20134): 194\u2013212. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F19386380903405090\" target=\"_blank\">10.1080\/19386380903405090<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Metadata+Best+Practice+for+a+Scientific+Data+Repository&rft.jtitle=Journal+of+Library+Metadata&rft.aulast=Greenberg%2C+J.%3B+White%2C+H.C.%3B+Carrier%2C+S.+et+al.&rft.au=Greenberg%2C+J.%3B+White%2C+H.C.%3B+Carrier%2C+S.+et+al.&rft.date=2009&rft.volume=9&rft.issue=3%E2%80%934&rft.pages=194%E2%80%93212&rft_id=info:doi\/10.1080%2F19386380903405090&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VUE-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VUE_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Vue Core Development Team. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/vuejs.org\/\" target=\"_blank\">\"Vue.js - The Progressive JavaScript Framework\"<\/a>. Evan You<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/vuejs.org\/\" target=\"_blank\">https:\/\/vuejs.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 25 May 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Vue.js+-+The+Progressive+JavaScript+Framework&rft.atitle=&rft.aulast=Vue+Core+Development+Team&rft.au=Vue+Core+Development+Team&rft.pub=Evan+You&rft_id=https%3A%2F%2Fvuejs.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Flask-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Flask_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/palletsprojects.com\/p\/flask\/\" target=\"_blank\">\"Flask\"<\/a>. The Pallets Project<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/palletsprojects.com\/p\/flask\/\" target=\"_blank\">https:\/\/palletsprojects.com\/p\/flask\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 25 May 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Flask&rft.atitle=&rft.pub=The+Pallets+Project&rft_id=https%3A%2F%2Fpalletsprojects.com%2Fp%2Fflask%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Django-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Django_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.djangoproject.com\/\" target=\"_blank\">\"Django\"<\/a>. The Django Software Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.djangoproject.com\/\" target=\"_blank\">https:\/\/www.djangoproject.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 25 May 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Django&rft.atitle=&rft.pub=The+Django+Software+Foundation&rft_id=https%3A%2F%2Fwww.djangoproject.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FieldingArch00-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FieldingArch00_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Fielding, R.T. (2000). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ics.uci.edu\/~fielding\/pubs\/dissertation\/top.htm\" target=\"_blank\">\"Architectural Styles and the Design of Network-based Software Architectures\"<\/a>. University of California, Irvine<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ics.uci.edu\/~fielding\/pubs\/dissertation\/top.htm\" target=\"_blank\">https:\/\/www.ics.uci.edu\/~fielding\/pubs\/dissertation\/top.htm<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Architectural+Styles+and+the+Design+of+Network-based+Software+Architectures&rft.atitle=&rft.aulast=Fielding%2C+R.T.&rft.au=Fielding%2C+R.T.&rft.date=2000&rft.pub=University+of+California%2C+Irvine&rft_id=https%3A%2F%2Fwww.ics.uci.edu%2F%7Efielding%2Fpubs%2Fdissertation%2Ftop.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OAIProt-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OAIProt_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.openarchives.org\/pmh\/\" target=\"_blank\">\"Open Archives Initiative Protocol for Metadata Harvesting\"<\/a>. Open Archives Initiative<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.openarchives.org\/pmh\/\" target=\"_blank\">http:\/\/www.openarchives.org\/pmh\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 25 May 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Open+Archives+Initiative+Protocol+for+Metadata+Harvesting&rft.atitle=&rft.pub=Open+Archives+Initiative&rft_id=http%3A%2F%2Fwww.openarchives.org%2Fpmh%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PostegreSQL-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PostegreSQL_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.postgresql.org\/\" target=\"_blank\">\"PostegreSQL: The World's Most Advanced Open Source Relational Database\"<\/a>. PostgreSQL Global Development Group<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.postgresql.org\/\" target=\"_blank\">https:\/\/www.postgresql.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 30 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=PostegreSQL%3A+The+World%27s+Most+Advanced+Open+Source+Relational+Database&rft.atitle=&rft.pub=PostgreSQL+Global+Development+Group&rft_id=https%3A%2F%2Fwww.postgresql.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Elasticsearch-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Elasticsearch_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.elastic.co\/elasticsearch\/\" target=\"_blank\">\"Elasticsearch: The Heart of the Free and Open Elastic Stack\"<\/a>. Elasticsearch B.V<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.elastic.co\/elasticsearch\/\" target=\"_blank\">https:\/\/www.elastic.co\/elasticsearch\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Elasticsearch%3A+The+Heart+of+the+Free+and+Open+Elastic+Stack&rft.atitle=&rft.pub=Elasticsearch+B.V&rft_id=https%3A%2F%2Fwww.elastic.co%2Felasticsearch%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZschummeIAM21-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZschummeIAM21_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Zschumme, P. (15 January 2021). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/zenodo.org\/record\/4442553\" target=\"_blank\">\"IAM-CMS\/process-manager (Version 0.1.0)\"<\/a>. <i>Zenodo<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5281%2Fzenodo.4442553\" target=\"_blank\">10.5281\/zenodo.4442553<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/zenodo.org\/record\/4442553\" target=\"_blank\">https:\/\/zenodo.org\/record\/4442553<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=IAM-CMS%2Fprocess-manager+%28Version+0.1.0%29&rft.atitle=Zenodo&rft.aulast=Zschumme%2C+P.&rft.au=Zschumme%2C+P.&rft.date=15+January+2021&rft_id=info:doi\/10.5281%2Fzenodo.4442553&rft_id=https%3A%2F%2Fzenodo.org%2Frecord%2F4442553&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZschummeIAMEng21-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZschummeIAMEng21_47-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Zschumme, P. (15 January 2021). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/zenodo.org\/record\/4442563\" target=\"_blank\">\"IAM-CMS\/process-engine (Version 0.1.0)\"<\/a>. <i>Zenodo<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5281%2Fzenodo.4442563\" target=\"_blank\">10.5281\/zenodo.4442563<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/zenodo.org\/record\/4442563\" target=\"_blank\">https:\/\/zenodo.org\/record\/4442563<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=IAM-CMS%2Fprocess-engine+%28Version+0.1.0%29&rft.atitle=Zenodo&rft.aulast=Zschumme%2C+P.&rft.au=Zschumme%2C+P.&rft.date=15+January+2021&rft_id=info:doi\/10.5281%2Fzenodo.4442563&rft_id=https%3A%2F%2Fzenodo.org%2Frecord%2F4442563&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WalkerShibb17-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WalkerShibb17_48-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Walker, D.; Cantor, S.; Carmody, S.; et al. (17 November 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/spaces.at.internet2.edu\/display\/TI\/TI.66.1\" target=\"_blank\">\"Shibboleth Architecture Protocols and Profiles\"<\/a>. <i>Internet2 Confluence<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/http%3A%2F%2Fdoi.org%2F10.26869%2FTI.66.1\" target=\"_blank\">http:\/\/doi.org\/10.26869\/TI.66.1<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/spaces.at.internet2.edu\/display\/TI\/TI.66.1\" target=\"_blank\">https:\/\/spaces.at.internet2.edu\/display\/TI\/TI.66.1<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Shibboleth+Architecture+Protocols+and+Profiles&rft.atitle=Internet2+Confluence&rft.aulast=Walker%2C+D.%3B+Cantor%2C+S.%3B+Carmody%2C+S.%3B+et+al.&rft.au=Walker%2C+D.%3B+Cantor%2C+S.%3B+Carmody%2C+S.%3B+et+al.&rft.date=17+November+2017&rft_id=info:doi\/http%3A%2F%2Fdoi.org%2F10.26869%2FTI.66.1&rft_id=https%3A%2F%2Fspaces.at.internet2.edu%2Fdisplay%2FTI%2FTI.66.1&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchoofIAM20-49\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchoofIAM20_49-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Schoof, E.; Brandt, N. (02 February 2021). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/zenodo.org\/record\/4088276\" target=\"_blank\">\"IAM-CMS\/kadi-apy: Kadi4Mat API Library (Version 0.4.1)\"<\/a>. <i>Zenodo<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5281%2Fzenodo.4507865\" target=\"_blank\">10.5281\/zenodo.4507865<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/zenodo.org\/record\/4088276\" target=\"_blank\">https:\/\/zenodo.org\/record\/4088276<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=IAM-CMS%2Fkadi-apy%3A+Kadi4Mat+API+Library+%28Version+0.4.1%29&rft.atitle=Zenodo&rft.aulast=Schoof%2C+E.%3B+Brandt%2C+N.&rft.au=Schoof%2C+E.%3B+Brandt%2C+N.&rft.date=02+February+2021&rft_id=info:doi\/10.5281%2Fzenodo.4507865&rft_id=https%3A%2F%2Fzenodo.org%2Frecord%2F4088276&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZschummeIAMNodes21-50\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZschummeIAMNodes21_50-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Zschumme, P. (16 October 2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/zenodo.org\/record\/4094719\" target=\"_blank\">\"IAM-CMS\/workflow-nodes (Version 0.1.0)\"<\/a>. <i>Zenodo<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5281%2Fzenodo.4094719\" target=\"_blank\">10.5281\/zenodo.4094719<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/zenodo.org\/record\/4094719\" target=\"_blank\">https:\/\/zenodo.org\/record\/4094719<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=IAM-CMS%2Fworkflow-nodes+%28Version+0.1.0%29&rft.atitle=Zenodo&rft.aulast=Zschumme%2C+P.&rft.au=Zschumme%2C+P.&rft.date=16+October+2020&rft_id=info:doi\/10.5281%2Fzenodo.4094719&rft_id=https%3A%2F%2Fzenodo.org%2Frecord%2F4094719&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchindelinTheImage15-51\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchindelinTheImage15_51-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schindelin, J.; Rueden, C.T.; Hiner, M.C. et al. (2015). \"The ImageJ ecosystem: An open platform for biomedical image analysis\". <i>Molecular Reproduction & Development<\/i> <b>82<\/b> (7\u20138): 518\u201329. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fmrd.22489\" target=\"_blank\">10.1002\/mrd.22489<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+ImageJ+ecosystem%3A+An+open+platform+for+biomedical+image+analysis&rft.jtitle=Molecular+Reproduction+%26+Development&rft.aulast=Schindelin%2C+J.%3B+Rueden%2C+C.T.%3B+Hiner%2C+M.C.+et+al.&rft.au=Schindelin%2C+J.%3B+Rueden%2C+C.T.%3B+Hiner%2C+M.C.+et+al.&rft.date=2015&rft.volume=82&rft.issue=7%E2%80%938&rft.pages=518%E2%80%9329&rft_id=info:doi\/10.1002%2Fmrd.22489&rfr_id=info:sid\/en.wikipedia.org:Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; however, this version lists them in order of appearance, by design.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194104\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.951 seconds\nReal time usage: 3.052 seconds\nPreprocessor visited node count: 35675\/1000000\nPreprocessor generated node count: 39753\/1000000\nPost\u2010expand include size: 224326\/2097152 bytes\nTemplate argument size: 73282\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 711.922 1 - -total\n 89.01% 633.662 1 - Template:Reflist\n 75.89% 540.304 51 - Template:Citation\/core\n 42.48% 302.442 26 - Template:Cite_journal\n 31.16% 221.857 22 - Template:Cite_web\n 6.04% 43.007 3 - Template:Cite_book\n 5.52% 39.330 1 - Template:Infobox_journal_article\n 5.28% 37.557 1 - Template:Infobox\n 5.10% 36.291 38 - Template:Citation\/identifier\n 4.16% 29.651 60 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12395-0!*!0!!en!5!* and timestamp 20210429194101 and revision id 41829\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science\">https:\/\/www.limswiki.org\/index.php\/Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","7d7600ccadb42ccd7bdbe90737c9955d_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/76\/Fig1_Brandt_DataSciJourn21_20-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/58\/Fig2_Brandt_DataSciJourn21_20-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0b\/Fig3_Brandt_DataSciJourn21_20-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/c3\/Fig4_Brandt_DataSciJourn21_20-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a6\/Fig5_Brandt_DataSciJourn21_20-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/6f\/Fig6_Brandt_DataSciJourn21_20-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/49\/Fig7_Brandt_DataSciJourn21_20-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/72\/Fig8_Brandt_DataSciJourn21_20-1.png"],"7d7600ccadb42ccd7bdbe90737c9955d_timestamp":1619725261,"bd8eee413799c826d26140a4bd9d594e_type":"article","bd8eee413799c826d26140a4bd9d594e_title":"Implement an international interoperable PHR by FHIR: A Taiwan innovative application (Lee et al. 2020)","bd8eee413799c826d26140a4bd9d594e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application","bd8eee413799c826d26140a4bd9d594e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Implement an international interoperable PHR by FHIR: A Taiwan innovative application\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nImplement an international interoperable PHR by FHIR: A Taiwan innovative applicationJournal\n \nSustainabilityAuthor(s)\n \nLee, Yen-Liang; Lee, Hsiu-An; Hsu, Chien-Yeh; Kung, Hsin-Yeh; Chiu, Hung-WenAuthor affiliation(s)\n \nTaipei Medical University, Chunghwa Telecom Laboratories, Tamkang University,\r\nSmart Healthcare Center of Excellence, National Taipei University of Nursing and Health SciencesPrimary contact\n \nhwchiu at tmu dot edu dot twYear published\n \n2020Volume and issue\n \n13(1)Article #\n \n198DOI\n \n10.3390\/su13010198ISSN\n \n2071-1050Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/2071-1050\/13\/1\/198\/htmDownload\n \nhttps:\/\/www.mdpi.com\/2071-1050\/13\/1\/198\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Literature review \n4 Materials and methods \n\n4.1 My Health Bank (MHB) content analysis \n4.2 FHIR resource correspondence \n4.3 Personal health record management platform design \n4.4 Authentication and security \n\n\n5 Results \n\n5.1 Interoperability PHR: Profile of MHB PHR based on FHIR format \n5.2 Data conversion of international format \n5.3 Personal health management platform \n5.4 Data verification mechanism \n\n\n6 Discussion \n7 Conclusion \n8 Acknowledgements \n\n8.1 Author contributions \n8.2 Funding \n8.3 Conflicts of interest \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nPersonal health records (PHRs) have many benefits for things such as health surveillance, epidemiological surveillance, self-control, links to various services, public health and health management, and international surveillance. The implementation of an international standard for interoperability is essential to accessing PHRs. In Taiwan, the nationwide exchange platform for electronic medical records (EMRs) has been in use for many years. The Health Level Seven International (HL7) Clinical Document Architecture (CDA) was used as the standard for those EMRs. However, the complication of implementing CDA became a barrier for many hospitals to realizing standard EMRs. \nIn this study, we implemented a Fast Healthcare Interoperability Resources (FHIR)-based PHR transformation process, including a user interface module to review the contents of PHRs. We used My Health Bank (MHB), a PHR data book developed and issued to all people by the Taiwan National Health Insurance, as the PHRs' contents in this study. Network Time Protocol (NTP)\/Simple Network Time Protocol (SNTP) was used in the security and user authentication mechanism when processing and applying personal health information. Transport Layer Security (TLS) 1.2 (such as HyperText Transfer Protocol Secure or HTTPS) was used for protection in data communication. User authentication is important in the platform. OAuth (OAuth 2.0) was used as a user authentication mechanism to confirm legitimate user access to ensure data security. The contents of MHB were analyzed and mapped to FHIR, and then converted to FHIR format according to the mapping logic template. The function of format conversion was carried out by using ASP.NET. XPath and JSPath technologies filtered out specific information tags. The converted data structure was verified through an HL7 application programming interface (HAPI) server, and a new JSON file was finally created. \nThis platform can not only capture any PHR based on the FHIR format but also publish FHIR-based MHB records to any other platform to bridge the interoperability gap between different PHR systems. Therefore, our implementation\/application with the automatic transformation from MHB to FHIR format provides an innovative method for people to access their own PHRs through MHB. No one has published a similar application like us using a nationwide PHR standard, MHB, in Taiwan. The application we developed will be very useful for a single person to use or for other system developers to implement their own standard PHR software.\nKeywords: FHIR, interoperability, PHR, data management, precision health management\n\nIntroduction \nPersonal health records (PHRs) are personalized records that include data related to health.[1] The Markle Foundation\u2019s \"Connecting for Health\" collaborative defines a PHR as \u201can electronic application through which individuals can access, manage, and share their health information, and that of others for whom they are authorized, in a private, secure, and confidential environment.\u201d[2] This differs from the more widely used electronic medical record (EMR), which focuses on clinical data and is operated by the medical service provider (such as clinics and hospitals). PHRs have great benefits for health monitoring, epidemiological surveillance, self-control, linkages with different services, and public health management in areas such as international health care.[3][4]\nRecently, medical services have placed more focus on precision medicine, which refers to specific medical treatments based on the individual characteristics of each patient.[5] PHRs are a crucial component of precision medicine. The information in PHRs can provide more details for clinical decision support. Based on a foundation of an electronic hospital information system (HIS), we can get plentiful personal health data to create PHRs. Taiwan has implemented national centralized health insurance information systems since the establishment of the Taiwan National Health Insurance Administration (NHIA) in 1995. According to CEOWORLD magazine\u2019s 2019 \u201cHealth Care Index\u201d statistics, Taiwan has the best medical system in the world[6], while the National Health Insurance (NHI) has a coverage rate of more than 99%.[7] The NHIA pays for most of the medical service expenses of all Taiwanese citizens to the hospitals and clinics that provide services. For the reimbursement procedure, hospitals and clinics need to upload patient care data related to the payment, including the diagnosis, prescriptions, treatment information, images, testing data, etc. to NHIA information systems. By doing so, the NHIA has collected almost all individual medical care data in Taiwan and stores it at the NHIA\u2019s data center. To promote personal health management, the NHIA launched the My Health Bank (MHB) system in September 2014. The MHB system is a personalized cloud-based service that aims to return personal medical data back to the citizens. Individuals can use their citizen digital certificate or password-registered NHI card as identity verification to download their medical data that have been collected by health insurance.\nThe NHIA aims to let citizens more directly control and manage their health data. Based on a cloud-based system, people can access their personal health insurance records in the past three years via the NHIA\u2019s MHB portal, and the data can be printed out or downloaded. The NHIA hopes that citizens will have access to MHB when they go to a clinic or hospital as a reference for the physician. This service promotes people to have their health information and know their own health status.\nHowever, despite the rich content of MHB and the data quality of the information confirmed in diagnosis, medicine, and other items, the information continues to use the original insurance declaration form and does not follow the relevant medical information standards. This makes it difficult to integrate and apply information. In addition, the official MHB system is mainly used as a data provider. The content is mainly based on the qualified medical insurance service records of various contracted hospitals, clinics, or institutions. Individuals are not allowed to add other personalized information themselves. Additionally, cloud-based systems can make it hard to share data with the doctor. When people see a doctor for an urgent reason, there may not be enough time to log into the MHB system and search for data. Although individuals can download data from the website, the data will be in the XML or JSON format, which most individuals cannot read. In addition, the content of PHRs does not follow any international format, making it quite difficult to exchange data or interoperate with other health information systems. The application and sharing of PHRs are very important. In order to achieve precision medicine, PHRs need to comply with international standards before they can be accepted, integrated, and applied by different systems.\nJung et al.[8] agree that an integrated personal health record is more valuable than a single record. The impact of interoperability on PHRs has been the focus of interdisciplinary researchers in recent years. A survey article by Alyami and Song[9] emphasized the vital role of interoperability in the implementation and adoption of PHRs. Plastiras and O'Sullivan[10] developed an ontology-driven intermediary layer to achieve interoperability between PHRs and electronic health records (EHRs) of various standards. Li[11] developed a service-oriented interoperable integrated PHR system and explained that it can be used to overcome interoperability problems between medical systems. Urbauer et al.[12] compared the interoperability procedures for the communication of medical systems and personal health devices to analyze the advantages and gaps, including presenting a total solution for interoperability. And Roehrs et al.[13] presented the crucial concept of interoperability for developing and using PHRs.\nAccording to other related studies, the key point of PHR applications from different sources is application integration and content standardization. In general, currently, most of the interoperability of medical information is still based on standardized data exchange and integration through the development of intermediary layers, data standard conversion, or ontology design. Taiwan has an advanced health insurance system and has collected complete personal health records since 1995. The huge health database gives Taiwan a good opportunity for precision medicine to use. However, due to the delayed process of government authority on the implementation of the data standardization and formatting, the data still cannot meet the requirements of international interoperability at present.\nIn Taiwan, the nationwide exchange platform for EMRs has been in use for many years. Although the Health Level Seven International (HL7) CDA was used as the standard of the EMRs, people outside the hospitals, including patients and many health management organizations, did not have the method to access the EMRs, and the complication of implementing CDA became a barrier for hospitals to realize the standard EMRs. As such, a concept was developed: the implementation with an automatic transformation from MHB to a Fast Healthcare Interoperability Resources (FHIR) format provides an innovative method for people to access their own PHRs through MHB. No one has published a similar application like us to use nationwide PHRs, i.e., MHB, in Taiwan. The application we developed will be very useful for a single person to use or for other system developers to implement their own standard PHR software.\nThis research focuses on converting Taiwan\u2019s personal health information into a standardized international standard format and provide a management platform for personal health records. Based on an innovative application model of personal data (MHB) unique to Taiwan\u2019s health insurance, we designed a demonstration framework to enable Taiwan to have better exemplary measures in personal health management, and take the lead in completing \u201cpatient-centered\u201d precise health management. The goal of this study was to develop an interoperable international personal health record using HL7 FHIR as the data standard and MHB data as the data content. This study designed a personal health management platform that could fit international standards and support users to manage personal health based on MHB. As long as the FHIR format health record is uploaded, people from different countries can use it all over the world to manage their personal health information, allow physicians to conduct precise medical treatment, and provide clinical decision-making support to users.\n\nLiterature review \nPersonal health information is an important foundation for the development of smart medical care. In addition to EMRs for medical consultations, institutional care records, and physiological data measured by medical materials, it also includes daily food content and wound conditions, which can be included in personal health records.\nAlthough there are many standards in the medical domain, the interoperability between different systems is still poor. There are many problems that need to be solved for system interoperability. Therefore, HL7 has developed FHIR as a new foundation to achieve interoperability. FHIR is currently a widely adopted health information standard. It is not a kind of EHR; rather, it describes the data format and data elements of EHRs. FHIR is the new HL7 standard that evolved from the HL7 v2, HL7 v3, and HL7 CDA standards, which aim for easier implementation.[14] The main goals of FHIR are to promote the effective communication and content of medical information and to widely use medical information on a variety of devices, including computers, tablets, IoT devices, and smartphones, so as to provide medical services to hospitals and individuals more easily. It also provides systematic information to third-party application developers. FHIR has the characteristics and value of being open and easy to expand by publicly providing data elements in a standardized format. It also provides an alternative to document-centric approaches by providing different data elements separately as a service. For example, with the corresponding resource URLs, basic healthcare service elements such as patients, admissions, diagnostic reports, and medications can be accessed and manipulated. The implementation of FHIR is based on the HL7 and HTTPS (HTTP Secure) protocols; therefore, the wire data analysis platform can parse messages for real-time data collection. According to this idea, when FHIR messages are transmitted over the network, the medical service organization will be able to use these messages to collect real-time data. This data can then be fed into the data store where it is associated with other informatics data. FHIR uses a web-based API suite, including HTML, HTTP-based RESTful protocols, and cascading style sheets (CSS) for the user interface. The document format can be based on the JSON or XML format.\nSchleyer et al. created a medical dashboard based on FHIR that integrates clinical information from the HISs with the EHR. Testing in a medical organization showed that the integration dashboard is useful. The clinical information can be integrated effectively based on FHIR.[15] FHIR uses widely adopted web technologies, is implementer-friendly and vendor-independent, and offers a useful architecture of applications and formats for EHR providers, healthcare providers, and public health professionals.[16] Baihan et al. implemented FHIR in the m-Health domain and used FHIR to integrate health information technology (HIT) systems with m-Health applications.[17] Daumke presented an architecture to harmonize commercial clinical text-mining tools by FHIR, which can be used as a clinical information model standard.[18] Recently, many research and industry products provide examples of how FHIR can be used for healthcare data integration [19][20][21][22][23][24][25][26], highlighting how FHIR has realized interoperability between different healthcare systems.\nHowever, Taiwan\u2019s MHB is not designed with reference to the good examples of other studies. Many adjustments are needed in terms of data compatibility, interoperability, and standardization.\nRoehrs et al.[27] proposed an application model to buffer the integration problems caused by the lack of interoperability between different standards. Medical organizations have also proposed various standards to alleviate interoperability problems, and convert some of the more popular standards into Unified Modeling Language (UML) to achieve model-driven development of medical and health applications.[28][29] Gonz\u00e1lez-Ferrer et al.[30] developed a PHR-based clinical decision support system, making PHR a core component of clinical decision support (CDS). After evaluation, the implementation was completed using openEHR and HL7 virtual medical records (vMRs). Simon et al. developed Ubiquitous PHR (uPHR)[31], which emphasizes data interoperability standards across different healthcare institutions. Marceglia et al.[32] designed a structured mobile PHR application. The application allows the use of HL7 CDA and an EHR (OpenEMR) for cloud sharing of health records to exchange information. Fonda et al.[33] developed a prototype personal health application (PHA) for patient-centered diabetes self-management. Under the Project Health Design program, a study[34] used \u201cObservation of Daily Life\u201d (ODL) to monitor personal health, such as PHRs for elderly people with arthritis and PHRs for self-assessment of asthma patients. Ferguson et al.[35] designed a personal health application for patients with heart failure. And Siek et al. developed the \u201cColorado Nursing Tablet\u201d[36] to help the elderly and their visiting caregivers manage medication plans during care.\nAccording to other integrated information technology architecture research, the interoperability of health data and data integration are the key points of development, which can effectively improve the availability of information systems and medical data, and provide convenient operating tools through an incapable architecture. Even though other studies have proposed prototypes of application systems, they still have some weaknesses. First, most systems are aimed at patients with specific diseases, such as Alzheimer\u2019s, heart disease, diabetes, etc., which reduces the interoperability and data integration. Second, the data are still based on specific disease groups and data content, and there is still a gap in follow-up treatment tracking and health management. Third, the patient\u2019s ability to manage and track data still needs to be strengthened.\n\nMaterials and methods \nThis research designed an FHIR-based personal health record adapter based on My Health Bank provided by the NHIA. Finally, an FHIR-compliant personal health management platform was established as a demonstration case. The research process of this research is shown in Figure 1.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Research process flow. FHIR = Fast Healthcare Interoperability Resources.\n\n\n\nMy Health Bank (MHB) content analysis \nMHB was launched by the NHIA in 2014. MHB is a personalized cloud-based service that aims to return personal medical data back to citizens. People can access their personal records and preventive health information from anywhere, and MHB data can be printed out or downloaded. However, the MHB system does not allow individuals to add other personalized information. The content is only claim-based medical data related to the National Health Insurance reimbursement process. Moreover, MHB does not comply with international standards. Although it is possible to get the data from the website, individuals still cannot read it themselves or exchange it with other organizations.\nMHB data includes the data owner\u2019s basic data and 11 other categories of data: outpatient data, inpatient data, dental data, allergy data, organ donation or palliative and hospice medical care choice, immunization information data, laboratory test data, imaging or pathological examination reports, traditional Chinese medicine data, and adult preventive data. The MHB data items are shown in Table 1. Different items have different content.\n\n\n\n\n\n\n\nTable 1. My Health Bank data content.\n\n\nCode\n\nData type\n\n\nbdata\n\nBasic information\n\n\nr1\n\nOutpatient data\n\n\nr2\n\nInpatient data\n\n\nr3\n\nDental data\n\n\nr4\n\nAllergy data\n\n\nr5\n\nOrgan donation or palliative and hospice medical care choice\n\n\nr6\n\nImmunization information data\n\n\nr7\n\nLaboratory test data\n\n\nr8\n\nImaging or pathological examination reports\n\n\nr9\n\nTraditional Chinese medicine data\n\n\nr10\n\nAdult preventative data\n\n\n\nThe content of the personal basic data includes national ID and name. Outpatient data, inpatient data, dental data, and traditional Chinese medicine data refer to clinical information and contain their own categories: medical institution details, medical date, disease diagnosis, personal payment amount, health insurance payment points, and prescription details. Allergy data can support physicians when prescribing medicine. Organ donation and palliative and hospice medical care choice data are part of MHB data, which the data owner decides in advance. When there is a note in this section, the patient\u2019s hospice medical behavior will be decided accordingly. The content of preventive inoculation data includes vaccination data, vaccination dates, and medical institution details. The content of laboratory test, imaging, or pathological examination reports, and screening data, which can improve medical service efficiency and reduce medical provider burden, include laboratory testing item or order details, result value, reference value, medical institution details, and image or pathology reports. Adult preventive data refer to preventive care and include height, weight, body mass index, waist circumference, blood pressure, cholesterol, blood sugar, kidney function, urine routine, metabolic syndrome, liver function, hepatitis B status, and hepatitis C status.\nDifferent items have different data presentation methods and structures. The MHB is converted to the FHIR format after comparing the code of the MHB with the PHR items.\nEach MHB has a total of three years of health insurance payment information. The information is presented as a single payment item by the unit. Figure 2 provides an example of a patient visit to a doctor on October 20, 2019 (20191020). Not that section b1.1 is personal national ID, b1.2 is the date of MHB create, r1.1 is the number of record, r1.2 is the clinic location, r1.3 is the ID of clinic, r1.4 is the name of clinic, r1.8 is the International Classification of Disease (ICD) code of diagnosis, r1.9 is the Chinese name of diagnosis, r1.12 is the payment of national health insurance, r1.13 is the co-payment of patient, and r1_1 is the medical and medication order of this diagnosis. This represents a slice of information for a single patient\u2019s medical treatment. In the MHB, the medical information for each medical visit for three years is distinguished and presented in different sections.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Example payment item from MHB.\n\n\n\nFHIR resource correspondence \nThe format of MHB is different to the structure that decomposes FHIR into separate resources. MHB does not correlate the contents of the data for each single visit. For example, when a patient goes to see a doctor for a fever, the clinical information (medical institution details, medical date, disease diagnosis, personal payment amount, health insurance payment points, and prescription details) will be stored in the same section. However, the information between the same section is not connected. The next time the patient goes to see the doctor, a new section will be created. During FHIR-format data storage, each piece of data needs to be concatenated and referenced to the other pieces of data. In the future, data search and management efficiency can greatly improve.\nThis study provided FHIR-based personal health records and designed a FHIR-based personal health management platform. FHIR \u201cBundle\u201d is used to combine whole data into one document. FHIR Bundle is a resource that gathers a collection of different resources into a single document. We used it to bundle resources together as a single health record.\nThe structure of the conversion of the health passbook into the FHIR format in this study is shown in Figure 3. The FHIR structure uses a three-tier design for format conversion. The first layer (black) uses the Bundle resource to integrate all the information of the MHB. The second layer (blue) uses a total of 11 different resources to store the detailed information of the MHB according to the information contained in the health passbook. The third layer (red) is the index between each data. In FHIR, the data will be scattered everywhere and then linked to each other by index to ensure that the data is complete and easy to retrieve.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. FHIR format PHR data structure. PHR = personal health record.\n\n\n\nA composition resource is a set of healthcare-related information that can connect all single pieces of information for one medical service and has clinical attestation.\nIn this study, composition resources were used to connect records with different pieces of data. The composition resources included patient information (reference to patient resources), medical institution details (reference to organization resources), medical service information (reference to encounter resources), disease diagnoses (reference to condition resources), personal payment amounts (reference to coverage resources), health insurance payment points (reference to coverage resources), treatment details (reference to procedure resources), prescription details (reference to medication request resources), and testing reports (reference to observation resources).\nThe FHIR Encounter resource was used to record the interaction between patient and healthcare provider. The data include healthcare services records and the medical service place.\nThe Encounter resource was used to record the information of each medical service, including clinical diagnoses, observations, medical procedures, and immunizations (reference to different resources). Due to the different structures of MHB and the FHIR format, it cannot connect the medication information to the Encounter resource directly. In our design, the Encounter resource was connected to the Composition resource for connecting all information.\nThe PHR from a different country should be converted to the same standard for cross-country interoperability PHR implementation. FHIR Release #4 (version 4.0.1) was implemented in this study.\n\nPersonal health record management platform design \nA platform was developed to view and transform PHRs. The platform contained two main modules. The architecture of the platform and module is shown in Figure 4. The MHB conversion module was used to convert the MHB data downloaded from the NHIA to the FHIR format. The index in the MHB was parsed and mapped to a separate FHIR resource. After data processing, an FHIR-format PHR could be created through this module. An HL7 application programming interface (HAPI) FHIR server was used as a health data server in our platform. HAPI FHIR is an open-source code that can fully implement the FHIR standard for interoperability in healthcare. It can store health data that comply with the FHIR format. Health data can be managed and transmitted based on a HAPI server, and HAPI can be used to verify the correctness of the FHIR format. If the format is not FHIR-compliant, the upload will be rejected.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Architecture of the platform and module. MHB = My Health Bank, RWD = real-world data, HAPI = Health Level Seven International Application Programming Interface.\n\n\n\nThe PHR viewer module could be used to view and manage personal health records, and PHR JSON files based on the FHIR format could be transferred by the viewer module.\nFor the design of data interoperability, the files after FHIR format conversion through the platform can be downloaded from the platform. The development of this research platform uses Windows 10 as the basic environment, and uses ASP.NET to set up a front visualization display platform and a back-end database data transmission and acceptance platform. The database uses a HAPI Server.\n\nAuthentication and security \nThe architecture of authentication and security mechanism in this platform is shown in Figure 5. Network Time Protocol (NTP)\/Simple Network Time Protocol (SNTP) was used in the security and user authentication mechanism when processing and applying personal health information. The correct time was sourced from the National Time and Frequency Standard Laboratory. Transport Layer Security (TLS) 1.2 (such as HTTP Secure (HTTPS)) was used for protection in the data communication part. TLS 1.2 was released in 2008 and can operate on HTTP\/2. Currently, mainstream browsers only implement encrypted HTTP\/2, which makes HTTP\/2 + TLS 1.2 or later a mandatory standard.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Authentication and security mechanism in this platform. NTP = Network Time Protocol, SNTP = Simple Network Time Protocol, TLS = Transport Layer Security.\n\n\n\nUser authentication is a critical part in this platform. Confirmation of legitimate user access can ensure data security. Referring to HL7 official recommendations, we used OAuth (OAuth 2.0) in the user authentication mechanism. OAuth2 is an authorization framework that allows users access to the user\u2019s private resources (such as photos, videos, and contact lists) stored on a certain website without a username and password through a third-party application. Third-party applications can obtain limited access to user accounts on OAuth2 providers (such as Microsoft, Google, Facebook, or Twitter) through HTTP\/HTTPS services.\nThis system does not verify the user itself (such as account\/password), but uses it in the OAuth2 environment where the OAuth2 provider is responsible for user authentication. Usually, when a user wants to log into the system, it will interactively redirect the user to the OAuth2 provider for authentication and obtain an OAuth2 token. Then, it sends back the HTTP request and provides the OAuth2 token in the HTTP Authentication header to represent the user. The system obtains the OAuth2 token and uses the OAuth2 authorization server to verify it and complete the login. In this way, system users can use accounts from Microsoft, Google, Facebook, or Twitter to log in with existing verification procedures, and only after logging into the system can they be associated with user data in the system.\n\nResults \nInteroperability PHR: Profile of MHB PHR based on FHIR format \nThe MHB PHRs based on FHIR format were implemented in this study. A total of 11 resources were included in our PHRs, including patients, composition, encounters, conditions, observations, procedures, medication requests, allergy intolerances, immunization recommendations, coverage, and organizations. The MHB data were converted to the FHIR format and stored in the personal health management platform.\nThe Bundle resource was used to connect the whole records. Different resources could be referenced through the subject element, which integrated the entire medical record of each medical service under the Composition resource. The Patient resource included personal information such as name, telephone number, gender, birthday, and address.\nEncounter resource data referred to medical service information, including diagnoses and service providers, and could be referenced to Condition, Procedure, Observation, Condition, Patient, and Organization. Condition resource data referred to diagnoses and could be referenced to Encounter and Patient. Observation data referred to testing reports, including type of testing and value, and could be referenced to Organization, Patient, and Encounter. Procedure referred to treatment details and could be referenced to Patient, Encounter, and Condition. Medication request data referred to prescription details, including the drug, dosage, and code, and could be referenced to Patient and Encounter. Allergy intolerance data referred to allergy information and could be referenced to Patient. Coverage data referred to payment information, including personal payment amounts and health insurance payment points, and could be referenced to Patient and Encounter. Organization data referred to organization information, including name, telephone number, and address. The structure of different resource connections is shown in Figure 6.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. Structure of different resource connections.\n\n\n\nIn Figure 6, each box represents a different resource. We converted the MHB single medical service record into a data format that conforms to the FHIR framework to meet international standards and subsequent data management applications.\nThe box representing each resource describes the relevant data that can be stored and the amount of data that can be stored. Take the composition resource on the left in Figure 5 as an example: it can contain different items of status, type, subject, encounter, date, author, title, and section. The field \u201c0..*\u201d means the amount of different data that can be stored. Take status as an example: the content of the field \u201c0..*\u201d is \u201c1..1\u201d, which means the status must have a data point and only can have one data point. Take section as another example: the content of the field \u201c0..*\u201d is \u201c0..*\u201d, which means that there can be no section data, or an unlimited number of sections (* means unlimited number).\n\nData conversion of international format \nThe conversion result of the MHB XML file into the FHIR standard of JSON format is shown in Figure 7. The different types of forms in the MHB were disassembled into different types of FHIR. According to the data format of each medical visit record in the MHB, the relevant data are referenced to the Composition resource. Each resource is also referenced to others.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 7. The convert result of the personal health record.\n\n\n\nFor example, the Encounter resource references all information about patient medical records, including resources such as Patient, Condition, Observation, Procedure, and Organization. Therefore, the information of each record can be retrieved from the Encounter resource. However, according to the FHIR standard, Encounter cannot reference medication and medical orders, so the Composition resource is used to concatenate the medication and treatment orders of each record to achieve the correlation between the data.\nThe main core of the data conversion module is the use of custom filters and conversion template technology. The data conversion module architecture is shown in Figure 8. Since the source data format of MHB can be XML or JSON, XPath and JSPath technologies are applied to filter out specific information modules. Then, we combined the corresponding defined MHB and FHIR format mapping logic template for conversion. In the template, items\/attributes, coding, and even logic rules are described for the information modules of the source and target. Through this model, whether it is a new version or a new item of MHB or a new version of FHIR in the future, resources can be easily mapped, and FHIR data (XML\/JSON) conforming to the latest specifications can be successfully produced.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 8. Data conversion module technology architecture.\n\n\n\nPersonal health management platform \nAfter the user uploads the MHB data to the platform, the PHRs will be automatically converted to the FHIR format. PHR data can be displayed in a platform for personal health management. An example of a personal health record and user interface is shown in Figure 9. Record examples include patient information, diagnoses, service provider organization information, medication information, and coverage. Health records in the FHIR format can be displayed and stored on the platform, and users can download PHRs in FHIR format for record management or data exchange. The platform has achieved international interoperability goals, and data exchange efficiency and high quality can be improved through the platform.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 9. An example of a personal health record and platform user interface.\n\n\n\nData verification mechanism \nAfter the data are converted, the correctness of the data format needs to be verified. A HAPI server is used in the data verification mechanism.\nHAPI is open-source software designed by a group of experts in the medical information field. Its architecture complies with FHIR standards and procedures. It has end-to-end code hygiene, secure defaults, regular updates, advanced features, reliability, predictable ownership, and in-house security features of expertise. After the HAPI server is established, data can be uploaded in an FHIR-compliant form.\nIn the data conversion process of this research, the converted FHIR JSON file was sent to the HAPI server to confirm that it is correct before the entire file was saved and output.\n\nDiscussion \nThe interoperability of the PHR format and international data exchange are essential in any kind of medical service. The interoperability of standards-based communication of system issue was previously analyzed by Urbauer et al.[12] They discussed the gaps in system interoperability communication and suggested possible solutions. Alyami et al.[9] and Roehrs et al.[13] proposed the key point of interoperability of personal health records. A service-oriented architecture for developing an interoperable integrated PHR system was presented by Li.[11] And Roehrs et al.[27] proposed an application model for integrating different standards in PHRs to solve the interoperability problem.\nAlthough many studies have discussed interoperability, it is inefficient to design a unique system for each different topic and goal. The HL7 FHIR achieved interoperability and system development efficiency by taking a different approach. Instead of using traditional data storage methods, HL7 FHIR uses a modular approach separating healthcare and medical service data (such as patient information, diagnoses, treatments, medications, and service records, etc.) into independent modules, which are called resources. The FHIR record control architecture is based on APIs and RESTful web services (the backbone of modern web applications) to achieve efficient and interoperable PHR control by utilizing modern web and mobile technology designs (such as HTTP-based lightweight REST protocol, JSON, or RDF, etc.).\nAccording to the research of Saripalle et al.[37], FHIR can be used as the basic structure of PHR and has the ability to customize the PHR. At present, Taiwan does not have any data that meet international standards and regulations, which leads to a gap in international medical treatment. In the era of globalization, data should be integrated and applied as soon as possible to improve the convenience of data exchange, access, and use.\nOn the other hand, under the application of data analysis and big-data-related artificial intelligence (AI), more and more attention is paid to data conforming to international standards, and only data conforming to standards can be quickly integrated and applied. Real-world evidence (RWE) research, for example, needs to ensure that key results of effectiveness can be observed and established in real-world data (RWD) sources, and a clinical RWD database can provide correct data to affect the overall research design and its clinical and cost burden. It may be necessary for a data repository to incorporate new data with the FHIR international format into the PHR or to collect and store biomarkers for subsequent analysis. In clinical research, studies such as the Salford Lung Study[38] have proven that the use of RWD data can make potential progress in generating RWE, but it has also found many challenges related to infrastructure and research funding resources.\n\nConclusion \nAlthough many studies have made clinical data management systems that meet international standards, most of them are based on patients with specific diseases, and data interoperability is still inflexible. The innovation of this research is to take personal health records as the main body of the contents and apply international standards to establish a demonstration platform. Through the integration of different technologies\u2014such as Xpath and JSPath for data conversion, HAPI server for data verification, and OAuth for authentication\u2014an interoperable platform with personal health records as the main body has been constructed.\nThis study leveraged HL7 FHIR to achieve interoperability as well as capture and share data using FHIR resources and formats. Taiwan\u2019s My Health Bank personal health records, launched by the NHI, was used as an international prototype. Our platform can perform personal health control and data management and provide PHRs based on FHIR. Users can exchange PHRs easily and efficiency. Based on the FHIR format, the platform can not only provide PHRs in the FHIR format but also capture any healthcare data based on FHIR. The main purposes of the platform were to demonstrate the content of HL7 FHIR resources and personal health information (profiles, health records, and functional standards) and to design and implement interoperable PHRs that are compatible with HL7 PHRs. The HL7 FHIR was a specification and standard, and the HAPI FHIR, which is a Java implementation of HL7 FHIR, was used as the database. The data captured in the PHR were structured as an FHIR resource and shared with the EHR in the JSON format using web services. The conversion of data was verified by HAPI in the correct format, and it was provided for users to download and view on website services. In general, this study allowed PHRs to be interoperable and easily exchanged and an architecture for international data exchange to be proposed.\nComparing our study to other studies, the degree of interoperability is higher and the applicable population is larger (not confined by specific health problems). There are very few restrictions to adopting our platform since we use HL7 FHIR and all other open-source resources, and the personal health management can be implemented since the MHB provides most of the information we need to do health management. Our platform provides a user-friendly authentication process based on the OAuth mechanism. Therefore, in this study, we have completed an innovative application framework in Taiwan. The application we developed will be very useful for a single person to use or for other system developers to implement their own standard PHR software.\n\nAcknowledgements \nAuthor contributions \nConceptualization, Y.-L.L., H.-A.L., and C.-Y.H.; methodology, Y.-L.L. and H.-A.L.; software, H.-H.K.; validation, H.-A.L.; system structure design, Y.-L.L. and H.-A.L.; writing\u2014original draft preparation, Y.-L.L. and H.-A.L.; writing\u2014review and editing, C.-Y.H. and H.-W.C.; supervision, H.-W.C. All authors have read and agreed to the published version of the manuscript.\n\nFunding \nThis research was funded by Ministry of Science and Technology, Taiwan grant number 109-2221-E-227 -003 -MY2.\n\nConflicts of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 Tang, P.C.; Ash, J.S.; Bates, D.W. et al. (2006). \"Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption\". JAMIA 13 (2): 121\u20136. doi:10.1197\/jamia.M2025. PMC PMC1447551. PMID 16357345. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447551 .   \n\n\u2191 Markle Foundation (01 July 2003). \"Connecting for Health: A Public-Private Collaborative\" (PDF). Markle Foundation. https:\/\/www.markle.org\/sites\/default\/files\/final_phwg_report1.pdf . Retrieved 10 March 2021 .   \n\n\u2191 Bower, J.K.; Bollinger, C.E.; Foraker, R.E. et al. (2017). \"Active Use of Electronic Health Records (EHRs) and Personal Health Records (PHRs) for Epidemiologic Research: Sample Representativeness and Nonresponse Bias in a Study of Women During Pregnancy\". EGEMS 5 (1): 1263. doi:10.13063\/2327-9214.1263. PMC PMC5340503. PMID 28303255. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5340503 .   \n\n\u2191 Bonanader, J.; Gates, S. (2010). \"Public health in an era of personal health records: Opportunities for innovation and new partnerships\". Journal of Medical Internet Research 12 (3): e33. doi:10.2196\/jmir.1346. PMC PMC2956336. PMID 20699216. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2956336 .   \n\n\u2191 Timmerman, L. (04 February 2013). \"What\u2019s in a Name? A Lot, When It Comes to \u2018Precision Medicine\u2019\". Xconomy. https:\/\/xconomy.com\/national\/2013\/02\/04\/whats-in-a-name-a-lot-when-it-comes-to-precision-medicine\/ .   \n\n\u2191 Ireland, S. (05 August 2019). \"Revealed: Countries With The Best Health Care Systems, 2019\". CEOWORLD magazine. https:\/\/ceoworld.biz\/2019\/08\/05\/revealed-countries-with-the-best-health-care-systems-2019\/ .   \n\n\u2191 Iqbal, U.; Nguyen, P.-A.; Syed-Abdul, S. et al.. Is long-term use of benzodiazepine a risk for cancer?. 94. e483. doi:10.1097\/MD.0000000000000483. PMC PMC4602739. PMID 25674736. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4602739 .   \n\n\u2191 Jung, S.Y.; Lee, K.; Hwang, H. et al. (2017). \"Support for Sustainable Use of Personal Health Records: Understanding the Needs of Users as a First Step Towards Patient-Driven Mobile Health\". JMIR mHealth and uHealth 5 (2): e19. doi:10.2196\/mhealth.6021. PMC PMC5344982. PMID 28232300. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5344982 .   \n\n\u2191 9.0 9.1 Alyami, M.A.; Song, Y.-T. (2016). \"Removing barriers in using personal health record systems\". Proceedings of the 2016 IEEE\/ACIS 15th International Conference on Computer and Information Science: 1\u20138. doi:10.1109\/ICIS.2016.7550810.   \n\n\u2191 Plastiras, P.; O'Sullivan, D.M. (2017). \"Combining Ontologies and Open Standards to Derive a Middle Layer Information Model for Interoperability of Personal and Electronic Health Records\". Journal of Medical Systems 41 (12): 195. doi:10.1007\/s10916-017-0838-9. PMID 29081012.   \n\n\u2191 11.0 11.1 Li, J. (2017). \"A Service-Oriented Approach to Interoperable and Secure Personal Health Record Systems\". Proceedings of the 2017 IEEE Symposium on Service-Oriented System Engineering. doi:10.1109\/SOSE.2017.20.   \n\n\u2191 12.0 12.1 Urbauer, P.; Sauermann, S.; Frohner, M. et al. (2015). \"Applicability of IHE\/Continua components for PHR systems: Learning from experiences\". Computers in Biology and Medicine 59: 186\u201393. doi:10.1016\/j.compbiomed.2013.12.003. PMID 24374230.   \n\n\u2191 13.0 13.1 Roehrs, A.; da Costa, C.A.; da Rosa Righi, R. et al. (2017). \"Personal Health Records: A Systematic Literature Review\". Journal of Medical Internet Research 19 (1): e13. doi:10.2196\/jmir.5876. PMC PMC5251169. PMID 28062391. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5251169 .   \n\n\u2191 \"Welcome to FHIR\". Health Level 7. 01 November 2019. https:\/\/hl7.org\/FHIR\/ . Retrieved 26 April 2020 .   \n\n\u2191 Schleyer, T.K.L.; Rahurkar, S.; Baublet, A.M. et al. (2019). \"Preliminary evaluation of the Chest Pain Dashboard, a FHIR-based approach for integrating health information exchange information directly into the clinical workflow\". AMIA Joint Summits on Translational Science 2019: 656-664. PMC PMC6568135. PMID 31259021. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6568135 .   \n\n\u2191 Goodman, A.B.; Braun, P.; Braunstein, M. (2018). \"Healthy Weight on FHIR - Innovative Technology to Support High Quality Clinical Care & Clinical to Community Linkages for Child Obesity\". PEdiatrics 141 (1 MeetingAbstract): 12. doi:10.1542\/peds.141.1_MeetingAbstract.12.   \n\n\u2191 Baihan, M.S.; S\u00e1nchez, Y.K.R.; Shao, X. et al. (2018). \"Chapter 6: A Blueprint for Designing and Developing M-Health Applications for Diverse Stakeholders Utilizing FHIR\". In Rajkumar, R.. Contemporary Applications of Mobile Computing in Healthcare Settings. IGI Global. pp. 85\u2013124. doi:10.4018\/978-1-5225-5036-5.ch006. ISBN 9781522550365.   \n\n\u2191 Daumke, P.; Heitmann, K.U.; Heckmann, S. et al. (2019). \"Clinical Text Mining on FHIR\". Studies in Health Technology and Informatics 264: 83\u20137. doi:10.3233\/SHTI190188. PMID 31437890.   \n\n\u2191 Storck, M.; Hollenberg, L.; Dugas, M. et al. (2019). \"Interoperability Improvement of Mobile Patient Survey (MoPat) Implementing Fast Health Interoperability Resources (FHIR)\". Studies in Health Technology and Informatics 258: 141\u201345. PMID 30942732.   \n\n\u2191 Giordanengo, A.; Bradway, M.; Gr\u00f8ttland, A. et al. (29 August 2017). \"A FHIR-based Data Flow Enabling Patients with Diabetes to Share Self-collected Data with the Norwegian National Healthcare Systems and Electronic Health Records Systems\" (PDF). 2017 Scandinavian Conference on Health Informatics. Norwegian Centre for E-health Research. https:\/\/ehealthresearch.no\/files\/documents\/Postere\/Poster_2018-02_16-9-ATTD-FHIR-based-Data-Flow.pdf .   \n\n\u2191 Kiourtis, A.; Mavrogiorgou, A.; Menychtas, A. et al. (2019). \"Structurally Mapping Healthcare Data to HL7 FHIR through Ontology Alignment\". Journal of Medical Systems 43 (3): 62. doi:10.1007\/s10916-019-1183-y. PMID 30721349.   \n\n\u2191 G\u00f8eg, K.R.; Rasmussen, R.K.; Jensen, L. et al. (2018). \"A future-proof architecture for telemedicine using loose-coupled modules and HL7 FHIR\". Computer Methods and Programs in Biomedicine 160: 95\u2013101. doi:10.1016\/j.cmpb.2018.03.010. PMID 29728251.   \n\n\u2191 Alper, B.; Mayer, M.; Shahin, K. et al. (2019). \"Achieving evidence interoperability in the computer age: Setting evidence on FHIR\". BMJ Evidence-Based Medicine 24 (Suppl. 1): A15. https:\/\/ebm.bmj.com\/content\/24\/Suppl_1\/A15.1 .   \n\n\u2191 Lackerbauer, A.M.; Lin, A.C.; Krauss, O. et al. (2018). \"A Model for Implementing an Interoperable Electronic Consent Form for Medical Treatment Using HL7 FHIR\". European Journal for Biomedical Informatics 14 (3): 37\u201347. doi:10.24105\/ejbi.2018.14.3.6.   \n\n\u2191 Dixon, B.E.; Taylor, D.E.; Choi, M. et al. (2019). \"Integration of FHIR to Facilitate Electronic Case Reporting: Results from a Pilot Study\". Studies in Health Technology and Informatics 264: 940\u201344. doi:10.3233\/SHTI190362. PMID 31438062.   \n\n\u2191 Hussain, M.A.; Langer, S.G.; Kohli, M. et al. (2018). \"Learning HL7 FHIR Using the HAPI FHIR Server and Its Use in Medical Imaging with the SIIM Dataset\". Journal of Digital Imaging 31 (3): 334\u201340. doi:10.1007\/s10278-018-0090-y. PMC PMC5959839. PMID 29725959. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5959839 .   \n\n\u2191 27.0 27.1 Roehrs, A.; da Costa, C.A.; da Rosa Righi, R. et al. (2019). \"Toward a Model for Personal Health Record Interoperability\". IEEE Journal of Biomedical and Health Informatics 23 (2): 867\u201373. doi:10.1109\/JBHI.2018.2836138. PMID 29993759.   \n\n\u2191 Mart\u00ednez-Garc\u00eda, A.; Garc\u00eda-Garc\u00eda, J.A.; Escalona, M.J. et al. (2015). \"Working with the HL7 metamodel in a Model Driven Engineering context\". Journal of Biomedical Informatics 57: 415\u201324. doi:10.1016\/j.jbi.2015.09.001. PMID 26348545.   \n\n\u2191 Walderhaug, S.; Stav, E.; Mikalsen, M. (2008). \"Experiences from Model-Driven Development of Homecare Services: UML Profiles and Domain Models\". Proceedings of MODELS 2008: International Conference on Model Driven Engineering Languages and Systems: 199\u2013212. doi:10.1007\/978-3-642-01648-6_22.   \n\n\u2191 Gonz\u00e1lez-Ferrer, A.; Peleg, M.; Verhees, B. et al. (2012). \"Data integration for clinical decision support based on open EHR archetypes and HL7 virtual medical record\". In Lenz, R.; Miksch, S.; Peleg, M. et al.. Process Support and Knowledge Representation in Health Care. Springer. pp. 71\u201384. doi:10.1007\/978-3-642-36438-9. ISBN 9783642364389.   \n\n\u2191 Simon, S.K.; Anbananthen, K.S.M.; Lee, S. (2013). \"A Ubiquitous Personal Health Record (uPHR) Framework\". Proceedings of the 2013 International Conference on Advanced Computer Science and Electronics Information: 423\u201327. doi:10.2991\/icacsei.2013.105.   \n\n\u2191 Marceglia, S.; Fontelo, P.; Rossi, E. et al. (2015). \"A Standards-Based Architecture Proposal for Integrating Patient mHealth Apps to Electronic Health Record Systems\". Applied Clinical Informatics 6 (3): 488-505. doi:10.4338\/ACI-2014-12-RA-0115. PMC PMC4586338. PMID 26448794. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4586338 .   \n\n\u2191 Fonda, S.J.; Kedziora, R.J.; Vigersky, R.A. et al. (2010). \"Evolution of a web-based, prototype Personal Health Application for diabetes self-management\". Journal of Biomedical Informatics 43 (5 Suppl.): S17\u201321. doi:10.1016\/j.jbi.2010.05.006. PMID 20937479.   \n\n\u2191 Backonja, U.; Kim, K.; Casper, G.R. et al. (2012). \"Observations of daily living: putting the \"personal\" in personal health records\". Proceedings of the 11th International Congress on Nursing Informatics 2012: 6. PMC PMC3799183. PMID 24199037. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3799183 .   \n\n\u2191 Ferguson, G.; Quinn, J.; Horwitz, C. et al. (2010). \"Towards a Personal Health Management Assistant\". Journal of Biomedical Informatics 43 (5 Suppl.): S13\u201316. doi:10.1016\/j.jbi.2010.05.007. PMID 20937478.   \n\n\u2191 Siek, K.A.; Ross, S.E.; Khan, D.U. et al. (2010). \"Colorado Care Tablet: The design of an interoperable Personal Health Application to help older adults with multimorbidity manage their medications\". Journal of Biomedical Informatics 43 (5 Suppl.): S22\u20136. doi:10.1016\/j.jbi.2010.05.014. PMID 20937480.   \n\n\u2191 Saripalle, R.; Runyan, C.; Russell, M. (2019). \"Using HL7 FHIR to achieve interoperability in patient health record\". Journal of Biomedical Informatics 94: 103188. doi:10.1016\/j.jbi.2019.103188. PMID 31063828.   \n\n\u2191 Bakerly, N.D.; Woodcock, A.; New, J.P. et al. (2015). \"The Salford Lung Study protocol: A pragmatic, randomised phase III real-world effectiveness trial in chronic obstructive pulmonary disease\". Respiratory Research 16 (1): 101. doi:10.1186\/s12931-015-0267-6. PMC PMC4558879. PMID 26337978. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4558879 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original reference 12 (Crabtree et al.) seems to have no bearing on the text and appears to have been accidentally included; it was omitted for this version. The same conclusion was reached about the original reference 38, which has also been omitted.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\">https:\/\/www.limswiki.org\/index.php\/Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on data management and sharingLIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 15 March 2021, at 18:40.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 509 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","bd8eee413799c826d26140a4bd9d594e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Implement_an_international_interoperable_PHR_by_FHIR_A_Taiwan_innovative_application skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Implement an international interoperable PHR by FHIR: A Taiwan innovative application<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Personal_health_record\" title=\"Personal health record\" class=\"wiki-link\" data-key=\"f79b1a737e567e2fcb4eb85a4835d853\">Personal health records<\/a> (PHRs) have many benefits for things such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Public_health_surveillance\" title=\"Public health surveillance\" class=\"wiki-link\" data-key=\"0467a7d2fe119533874d1108534438f6\">health surveillance<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Epidemiology\" title=\"Epidemiology\" class=\"wiki-link\" data-key=\"123badb8bf0b37a513182dbcfc3875bc\">epidemiological surveillance<\/a>, self-control, links to various services, <a href=\"https:\/\/www.limswiki.org\/index.php\/Public_health\" title=\"Public health\" class=\"wiki-link\" data-key=\"81092e25c0bd359cedd1b9f9dc350c86\">public health<\/a> and health management, and international surveillance. The implementation of an international standard for interoperability is essential to accessing PHRs. In Taiwan, the nationwide exchange platform for <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical records<\/a> (EMRs) has been in use for many years. The <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"e0bf845fb58d2bae05a846b47629e86f\">Health Level Seven International<\/a> (HL7) Clinical Document Architecture (CDA) was used as the standard for those EMRs. However, the complication of implementing CDA became a barrier for many <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospitals<\/a> to realizing standard EMRs. \n<\/p><p>In this study, we implemented a <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7#Fast_Healthcare_Interoperability_Resources_.28FHIR.29\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"ce348e672e47598e4b42cbd03c292ec4\">Fast Healthcare Interoperability Resources<\/a> (FHIR)-based PHR transformation process, including a user interface module to review the contents of PHRs. We used My Health Bank (MHB), a PHR data book developed and issued to all people by the Taiwan National Health Insurance, as the PHRs' contents in this study. Network Time Protocol (NTP)\/Simple Network Time Protocol (SNTP) was used in the security and user authentication mechanism when processing and applying <a href=\"https:\/\/www.limswiki.org\/index.php\/Protected_health_information\" title=\"Protected health information\" class=\"wiki-link\" data-key=\"eca2f6661b6896668bd523e640e12499\">personal health information<\/a>. Transport Layer Security (TLS) 1.2 (such as HyperText Transfer Protocol Secure or HTTPS) was used for protection in data communication. User authentication is important in the platform. OAuth (OAuth 2.0) was used as a user authentication mechanism to confirm legitimate user access to ensure <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_security\" title=\"Information security\" class=\"wiki-link\" data-key=\"9eff362d944224ff1d4ffe3a149d7cff\">data security<\/a>. The contents of MHB were analyzed and mapped to FHIR, and then converted to FHIR format according to the mapping logic template. The function of format conversion was carried out by using ASP.NET. XPath and JSPath technologies filtered out specific <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> tags. The converted data structure was verified through an HL7 <a href=\"https:\/\/www.limswiki.org\/index.php\/Application_programming_interface\" title=\"Application programming interface\" class=\"wiki-link\" data-key=\"36fc319869eba4613cb0854b421b0934\">application programming interface<\/a> (HAPI) server, and a new JSON file was finally created. \n<\/p><p>This platform can not only capture any PHR based on the FHIR format but also publish FHIR-based MHB records to any other platform to bridge the interoperability gap between different PHR systems. Therefore, our implementation\/application with the automatic transformation from MHB to FHIR format provides an innovative method for people to access their own PHRs through MHB. No one has published a similar application like us using a nationwide PHR standard, MHB, in Taiwan. The application we developed will be very useful for a single person to use or for other system developers to implement their own standard PHR software.\n<\/p><p><b>Keywords<\/b>: FHIR, interoperability, PHR, data management, precision health management\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Personal_health_record\" title=\"Personal health record\" class=\"wiki-link\" data-key=\"f79b1a737e567e2fcb4eb85a4835d853\">Personal health records<\/a> (PHRs) are personalized records that include data related to health.<sup id=\"rdp-ebb-cite_ref-TangPerson06_1-0\" class=\"reference\"><a href=\"#cite_note-TangPerson06-1\">[1]<\/a><\/sup> The Markle Foundation\u2019s \"Connecting for Health\" collaborative defines a PHR as \u201can electronic application through which individuals can access, manage, and share their health information, and that of others for whom they are authorized, in a private, secure, and confidential environment.\u201d<sup id=\"rdp-ebb-cite_ref-MarkleConnect03_2-0\" class=\"reference\"><a href=\"#cite_note-MarkleConnect03-2\">[2]<\/a><\/sup> This differs from the more widely used <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical record<\/a> (EMR), which focuses on clinical data and is operated by the medical service provider (such as clinics and <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospitals<\/a>). PHRs have great benefits for health monitoring, <a href=\"https:\/\/www.limswiki.org\/index.php\/Epidemiology\" title=\"Epidemiology\" class=\"wiki-link\" data-key=\"123badb8bf0b37a513182dbcfc3875bc\">epidemiological surveillance<\/a>, self-control, linkages with different services, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Public_health\" title=\"Public health\" class=\"wiki-link\" data-key=\"81092e25c0bd359cedd1b9f9dc350c86\">public health<\/a> management in areas such as international health care.<sup id=\"rdp-ebb-cite_ref-PowerActive17_3-0\" class=\"reference\"><a href=\"#cite_note-PowerActive17-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BonanderPublic10_4-0\" class=\"reference\"><a href=\"#cite_note-BonanderPublic10-4\">[4]<\/a><\/sup>\n<\/p><p>Recently, medical services have placed more focus on precision medicine, which refers to specific medical treatments based on the individual characteristics of each patient.<sup id=\"rdp-ebb-cite_ref-TimmermanWhat13_5-0\" class=\"reference\"><a href=\"#cite_note-TimmermanWhat13-5\">[5]<\/a><\/sup> PHRs are a crucial component of precision medicine. The <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> in PHRs can provide more details for clinical decision support. Based on a foundation of an electronic <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital_information_system\" title=\"Hospital information system\" class=\"wiki-link\" data-key=\"d8385de7b1f39a39d793f8ce349b448d\">hospital information system<\/a> (HIS), we can get plentiful personal health data to create PHRs. Taiwan has implemented national centralized health insurance information systems since the establishment of the Taiwan National Health Insurance Administration (NHIA) in 1995. According to <i>CEOWORLD magazine<\/i>\u2019s 2019 \u201cHealth Care Index\u201d statistics, Taiwan has the best medical system in the world<sup id=\"rdp-ebb-cite_ref-IrelandReveal19_6-0\" class=\"reference\"><a href=\"#cite_note-IrelandReveal19-6\">[6]<\/a><\/sup>, while the National Health Insurance (NHI) has a coverage rate of more than 99%.<sup id=\"rdp-ebb-cite_ref-IqbalIsLong15_7-0\" class=\"reference\"><a href=\"#cite_note-IqbalIsLong15-7\">[7]<\/a><\/sup> The NHIA pays for most of the medical service expenses of all Taiwanese citizens to the hospitals and clinics that provide services. For the reimbursement procedure, hospitals and clinics need to upload patient care data related to the payment, including the diagnosis, prescriptions, treatment information, images, testing data, etc. to NHIA information systems. By doing so, the NHIA has collected almost all individual medical care data in Taiwan and stores it at the NHIA\u2019s data center. To promote personal health management, the NHIA launched the My Health Bank (MHB) system in September 2014. The MHB system is a personalized <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud-based<\/a> service that aims to return personal medical data back to the citizens. Individuals can use their citizen digital certificate or password-registered NHI card as identity verification to download their medical data that have been collected by health insurance.\n<\/p><p>The NHIA aims to let citizens more directly control and manage their health data. Based on a cloud-based system, people can access their personal health insurance records in the past three years via the NHIA\u2019s MHB portal, and the data can be printed out or downloaded. The NHIA hopes that citizens will have access to MHB when they go to a clinic or hospital as a reference for the physician. This service promotes people to have their health information and know their own health status.\n<\/p><p>However, despite the rich content of MHB and the data quality of the information confirmed in diagnosis, medicine, and other items, the information continues to use the original insurance declaration form and does not follow the relevant medical information standards. This makes it difficult to integrate and apply information. In addition, the official MHB system is mainly used as a data provider. The content is mainly based on the qualified medical insurance service records of various contracted hospitals, clinics, or institutions. Individuals are not allowed to add other personalized information themselves. Additionally, cloud-based systems can make it hard to share data with the doctor. When people see a doctor for an urgent reason, there may not be enough time to log into the MHB system and search for data. Although individuals can download data from the website, the data will be in the XML or JSON format, which most individuals cannot read. In addition, the content of PHRs does not follow any international format, making it quite difficult to exchange data or interoperate with other <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">health information systems<\/a>. The application and sharing of PHRs are very important. In order to achieve precision medicine, PHRs need to comply with international standards before they can be accepted, integrated, and applied by different systems.\n<\/p><p>Jung <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-JungSupport17_8-0\" class=\"reference\"><a href=\"#cite_note-JungSupport17-8\">[8]<\/a><\/sup> agree that an integrated personal health record is more valuable than a single record. The impact of interoperability on PHRs has been the focus of interdisciplinary researchers in recent years. A survey article by Alyami and Song<sup id=\"rdp-ebb-cite_ref-AlyamiRemov16_9-0\" class=\"reference\"><a href=\"#cite_note-AlyamiRemov16-9\">[9]<\/a><\/sup> emphasized the vital role of interoperability in the implementation and adoption of PHRs. Plastiras and O'Sullivan<sup id=\"rdp-ebb-cite_ref-PlastirasCombin17_10-0\" class=\"reference\"><a href=\"#cite_note-PlastirasCombin17-10\">[10]<\/a><\/sup> developed an <a href=\"https:\/\/www.limswiki.org\/index.php\/Ontology_(information_science)\" title=\"Ontology (information science)\" class=\"wiki-link\" data-key=\"52d0664bde4b458e81fbc128b911a4a6\">ontology<\/a>-driven intermediary layer to achieve interoperability between PHRs and <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a> (EHRs) of various standards. Li<sup id=\"rdp-ebb-cite_ref-LiAServ17_11-0\" class=\"reference\"><a href=\"#cite_note-LiAServ17-11\">[11]<\/a><\/sup> developed a service-oriented interoperable integrated PHR system and explained that it can be used to overcome interoperability problems between medical systems. Urbauer <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-UrbauerAppl15_12-0\" class=\"reference\"><a href=\"#cite_note-UrbauerAppl15-12\">[12]<\/a><\/sup> compared the interoperability procedures for the communication of medical systems and personal health devices to analyze the advantages and gaps, including presenting a total solution for interoperability. And Roehrs <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-RoehrsPerson17_13-0\" class=\"reference\"><a href=\"#cite_note-RoehrsPerson17-13\">[13]<\/a><\/sup> presented the crucial concept of interoperability for developing and using PHRs.\n<\/p><p>According to other related studies, the key point of PHR applications from different sources is application integration and content standardization. In general, currently, most of the interoperability of medical information is still based on standardized data exchange and integration through the development of intermediary layers, data standard conversion, or ontology design. Taiwan has an advanced health insurance system and has collected complete personal health records since 1995. The huge health database gives Taiwan a good opportunity for precision medicine to use. However, due to the delayed process of government authority on the implementation of the data standardization and formatting, the data still cannot meet the requirements of international interoperability at present.\n<\/p><p>In Taiwan, the nationwide exchange platform for EMRs has been in use for many years. Although the <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"e0bf845fb58d2bae05a846b47629e86f\">Health Level Seven International<\/a> (HL7) CDA was used as the standard of the EMRs, people outside the hospitals, including patients and many health management organizations, did not have the method to access the EMRs, and the complication of implementing CDA became a barrier for hospitals to realize the standard EMRs. As such, a concept was developed: the implementation with an automatic transformation from MHB to a <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7#Fast_Healthcare_Interoperability_Resources_.28FHIR.29\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"ce348e672e47598e4b42cbd03c292ec4\">Fast Healthcare Interoperability Resources<\/a> (FHIR) format provides an innovative method for people to access their own PHRs through MHB. No one has published a similar application like us to use nationwide PHRs, i.e., MHB, in Taiwan. The application we developed will be very useful for a single person to use or for other system developers to implement their own standard PHR software.\n<\/p><p>This research focuses on converting Taiwan\u2019s personal health information into a standardized international standard format and provide a management platform for personal health records. Based on an innovative application model of personal data (MHB) unique to Taiwan\u2019s health insurance, we designed a demonstration framework to enable Taiwan to have better exemplary measures in personal health management, and take the lead in completing \u201cpatient-centered\u201d precise health management. The goal of this study was to develop an interoperable international personal health record using HL7 FHIR as the data standard and MHB data as the data content. This study designed a personal health management platform that could fit international standards and support users to manage personal health based on MHB. As long as the FHIR format health record is uploaded, people from different countries can use it all over the world to manage their <a href=\"https:\/\/www.limswiki.org\/index.php\/Protected_health_information\" title=\"Protected health information\" class=\"wiki-link\" data-key=\"eca2f6661b6896668bd523e640e12499\">personal health information<\/a>, allow physicians to conduct precise medical treatment, and provide clinical decision-making support to users.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Literature_review\">Literature review<\/span><\/h2>\n<p>Personal health information is an important foundation for the development of smart medical care. In addition to EMRs for medical consultations, institutional care records, and physiological data measured by medical materials, it also includes daily food content and wound conditions, which can be included in personal health records.\n<\/p><p>Although there are many standards in the medical domain, the interoperability between different systems is still poor. There are many problems that need to be solved for system interoperability. Therefore, HL7 has developed FHIR as a new foundation to achieve interoperability. FHIR is currently a widely adopted health information standard. It is not a kind of EHR; rather, it describes the data format and data elements of EHRs. FHIR is the new HL7 standard that evolved from the HL7 v2, HL7 v3, and HL7 CDA standards, which aim for easier implementation.<sup id=\"rdp-ebb-cite_ref-HL7_FHIR_14-0\" class=\"reference\"><a href=\"#cite_note-HL7_FHIR-14\">[14]<\/a><\/sup> The main goals of FHIR are to promote the effective communication and content of medical information and to widely use medical information on a variety of devices, including computers, tablets, IoT devices, and smartphones, so as to provide medical services to hospitals and individuals more easily. It also provides systematic information to third-party application developers. FHIR has the characteristics and value of being open and easy to expand by publicly providing data elements in a standardized format. It also provides an alternative to document-centric approaches by providing different data elements separately as a service. For example, with the corresponding resource URLs, basic healthcare service elements such as patients, admissions, diagnostic reports, and medications can be accessed and manipulated. The implementation of FHIR is based on the HL7 and HTTPS (HTTP Secure) protocols; therefore, the wire data analysis platform can parse messages for real-time data collection. According to this idea, when FHIR messages are transmitted over the network, the medical service organization will be able to use these messages to collect real-time data. This data can then be fed into the data store where it is associated with other informatics data. FHIR uses a web-based API suite, including HTML, HTTP-based RESTful protocols, and cascading style sheets (CSS) for the user interface. The document format can be based on the JSON or XML format.\n<\/p><p>Schleyer <i>et al.<\/i> created a medical dashboard based on FHIR that integrates clinical information from the HISs with the EHR. Testing in a medical organization showed that the integration dashboard is useful. The clinical information can be integrated effectively based on FHIR.<sup id=\"rdp-ebb-cite_ref-SchleyerPrelim19_15-0\" class=\"reference\"><a href=\"#cite_note-SchleyerPrelim19-15\">[15]<\/a><\/sup> FHIR uses widely adopted web technologies, is implementer-friendly and vendor-independent, and offers a useful architecture of applications and formats for EHR providers, healthcare providers, and public health professionals.<sup id=\"rdp-ebb-cite_ref-GoodmanHealthy18_16-0\" class=\"reference\"><a href=\"#cite_note-GoodmanHealthy18-16\">[16]<\/a><\/sup> Baihan <i>et al.<\/i> implemented FHIR in the m-Health domain and used FHIR to integrate health information technology (HIT) systems with m-Health applications.<sup id=\"rdp-ebb-cite_ref-BaihanABlue18_17-0\" class=\"reference\"><a href=\"#cite_note-BaihanABlue18-17\">[17]<\/a><\/sup> Daumke presented an architecture to harmonize commercial clinical text-mining tools by FHIR, which can be used as a clinical information model standard.<sup id=\"rdp-ebb-cite_ref-DaumkeClinical19_18-0\" class=\"reference\"><a href=\"#cite_note-DaumkeClinical19-18\">[18]<\/a><\/sup> Recently, many research and industry products provide examples of how FHIR can be used for healthcare data integration <sup id=\"rdp-ebb-cite_ref-StorckInter19_19-0\" class=\"reference\"><a href=\"#cite_note-StorckInter19-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GiordanengoAFHIR18_20-0\" class=\"reference\"><a href=\"#cite_note-GiordanengoAFHIR18-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KiourtisStruct19_21-0\" class=\"reference\"><a href=\"#cite_note-KiourtisStruct19-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-G.C3.B8egAFutur18_22-0\" class=\"reference\"><a href=\"#cite_note-G.C3.B8egAFutur18-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlperAchiev19_23-0\" class=\"reference\"><a href=\"#cite_note-AlperAchiev19-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LackerbauerAModel18_24-0\" class=\"reference\"><a href=\"#cite_note-LackerbauerAModel18-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DixonInteg19_25-0\" class=\"reference\"><a href=\"#cite_note-DixonInteg19-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HussainLearn18_26-0\" class=\"reference\"><a href=\"#cite_note-HussainLearn18-26\">[26]<\/a><\/sup>, highlighting how FHIR has realized interoperability between different healthcare systems.\n<\/p><p>However, Taiwan\u2019s MHB is not designed with reference to the good examples of other studies. Many adjustments are needed in terms of data compatibility, interoperability, and standardization.\n<\/p><p>Roehrs <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-RoehrsToward19_27-0\" class=\"reference\"><a href=\"#cite_note-RoehrsToward19-27\">[27]<\/a><\/sup> proposed an application model to buffer the integration problems caused by the lack of interoperability between different standards. Medical organizations have also proposed various standards to alleviate interoperability problems, and convert some of the more popular standards into Unified Modeling Language (UML) to achieve model-driven development of medical and health applications.<sup id=\"rdp-ebb-cite_ref-Mart.C3.ADnez-Garc.C3.ADaWork15_28-0\" class=\"reference\"><a href=\"#cite_note-Mart.C3.ADnez-Garc.C3.ADaWork15-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WalderhaugExper08_29-0\" class=\"reference\"><a href=\"#cite_note-WalderhaugExper08-29\">[29]<\/a><\/sup> Gonz\u00e1lez-Ferrer <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Gonz.C3.A1lez-FerrerData12_30-0\" class=\"reference\"><a href=\"#cite_note-Gonz.C3.A1lez-FerrerData12-30\">[30]<\/a><\/sup> developed a PHR-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support system<\/a>, making PHR a core component of clinical decision support (CDS). After evaluation, the implementation was completed using <a href=\"https:\/\/www.limswiki.org\/index.php\/OpenEHR\" title=\"OpenEHR\" class=\"wiki-link\" data-key=\"4c3f67f4e3102639ce8c6bff842b8982\">openEHR<\/a> and HL7 virtual medical records (vMRs). Simon <i>et al.<\/i> developed Ubiquitous PHR (uPHR)<sup id=\"rdp-ebb-cite_ref-SimonAUbiq13_31-0\" class=\"reference\"><a href=\"#cite_note-SimonAUbiq13-31\">[31]<\/a><\/sup>, which emphasizes data interoperability standards across different healthcare institutions. Marceglia <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MarcegliaAStand15_32-0\" class=\"reference\"><a href=\"#cite_note-MarcegliaAStand15-32\">[32]<\/a><\/sup> designed a structured mobile PHR application. The application allows the use of HL7 CDA and an EHR (<a href=\"https:\/\/www.limswiki.org\/index.php\/OpenEMR\" title=\"OpenEMR\" class=\"wiki-link\" data-key=\"ad61bf020caf7f6dd1c1ea1e59c9fe9a\">OpenEMR<\/a>) for cloud sharing of health records to exchange information. Fonda <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-FondaEvol10_33-0\" class=\"reference\"><a href=\"#cite_note-FondaEvol10-33\">[33]<\/a><\/sup> developed a prototype personal health application (PHA) for patient-centered diabetes self-management. Under the Project Health Design program, a study<sup id=\"rdp-ebb-cite_ref-BackonjaObserv12_34-0\" class=\"reference\"><a href=\"#cite_note-BackonjaObserv12-34\">[34]<\/a><\/sup> used \u201cObservation of Daily Life\u201d (ODL) to monitor personal health, such as PHRs for elderly people with arthritis and PHRs for self-assessment of asthma patients. Ferguson <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-FergusonTowards10_35-0\" class=\"reference\"><a href=\"#cite_note-FergusonTowards10-35\">[35]<\/a><\/sup> designed a personal health application for patients with heart failure. And Siek <i>et al.<\/i> developed the \u201cColorado Nursing Tablet\u201d<sup id=\"rdp-ebb-cite_ref-SiekColor10_36-0\" class=\"reference\"><a href=\"#cite_note-SiekColor10-36\">[36]<\/a><\/sup> to help the elderly and their visiting caregivers manage medication plans during care.\n<\/p><p>According to other integrated information technology architecture research, the interoperability of health data and data integration are the key points of development, which can effectively improve the availability of information systems and medical data, and provide convenient operating tools through an incapable architecture. Even though other studies have proposed prototypes of application systems, they still have some weaknesses. First, most systems are aimed at patients with specific diseases, such as Alzheimer\u2019s, heart disease, diabetes, etc., which reduces the interoperability and data integration. Second, the data are still based on specific disease groups and data content, and there is still a gap in follow-up treatment tracking and health management. Third, the patient\u2019s ability to manage and track data still needs to be strengthened.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<p>This research designed an FHIR-based personal health record adapter based on My Health Bank provided by the NHIA. Finally, an FHIR-compliant personal health management platform was established as a demonstration case. The research process of this research is shown in Figure 1.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"ba6ba4712bccd770da8eef7783212e4a\"><img alt=\"Fig1 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b3\/Fig1_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Research process flow. FHIR = Fast Healthcare Interoperability Resources.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"My_Health_Bank_.28MHB.29_content_analysis\">My Health Bank (MHB) content analysis<\/span><\/h3>\n<p>MHB was launched by the NHIA in 2014. MHB is a personalized cloud-based service that aims to return personal medical data back to citizens. People can access their personal records and preventive health information from anywhere, and MHB data can be printed out or downloaded. However, the MHB system does not allow individuals to add other personalized information. The content is only claim-based medical data related to the National Health Insurance reimbursement process. Moreover, MHB does not comply with international standards. Although it is possible to get the data from the website, individuals still cannot read it themselves or exchange it with other organizations.\n<\/p><p>MHB data includes the data owner\u2019s basic data and 11 other categories of data: outpatient data, inpatient data, dental data, allergy data, organ donation or palliative and hospice medical care choice, immunization information data, laboratory test data, imaging or pathological examination reports, traditional Chinese medicine data, and adult preventive data. The MHB data items are shown in Table 1. Different items have different content.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> My Health Bank data content.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Code\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Data type\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">bdata\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Basic information\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Outpatient data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Inpatient data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dental data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Allergy data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Organ donation or palliative and hospice medical care choice\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Immunization information data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Laboratory test data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Imaging or pathological examination reports\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Traditional Chinese medicine data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">r10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Adult preventative data\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The content of the personal basic data includes national ID and name. Outpatient data, inpatient data, dental data, and traditional Chinese medicine data refer to clinical information and contain their own categories: medical institution details, medical date, disease diagnosis, personal payment amount, health insurance payment points, and prescription details. Allergy data can support physicians when prescribing medicine. Organ donation and palliative and hospice medical care choice data are part of MHB data, which the data owner decides in advance. When there is a note in this section, the patient\u2019s hospice medical behavior will be decided accordingly. The content of preventive inoculation data includes vaccination data, vaccination dates, and medical institution details. The content of laboratory test, imaging, or pathological examination reports, and screening data, which can improve medical service efficiency and reduce medical provider burden, include laboratory testing item or order details, result value, reference value, medical institution details, and image or pathology reports. Adult preventive data refer to preventive care and include height, weight, body mass index, waist circumference, blood pressure, cholesterol, blood sugar, kidney function, urine routine, metabolic syndrome, liver function, hepatitis B status, and hepatitis C status.\n<\/p><p>Different items have different data presentation methods and structures. The MHB is converted to the FHIR format after comparing the code of the MHB with the PHR items.\n<\/p><p>Each MHB has a total of three years of health insurance payment information. The information is presented as a single payment item by the unit. Figure 2 provides an example of a patient visit to a doctor on October 20, 2019 (20191020). Not that section b1.1 is personal national ID, b1.2 is the date of MHB create, r1.1 is the number of record, r1.2 is the clinic location, r1.3 is the ID of clinic, r1.4 is the name of clinic, r1.8 is the International Classification of Disease (ICD) code of diagnosis, r1.9 is the Chinese name of diagnosis, r1.12 is the payment of national health insurance, r1.13 is the co-payment of patient, and r1_1 is the medical and medication order of this diagnosis. This represents a slice of information for a single patient\u2019s medical treatment. In the MHB, the medical information for each medical visit for three years is distinguished and presented in different sections.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"4960d80746af0788506efb002f077e13\"><img alt=\"Fig2 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/77\/Fig2_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Example payment item from MHB.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"FHIR_resource_correspondence\">FHIR resource correspondence<\/span><\/h3>\n<p>The format of MHB is different to the structure that decomposes FHIR into separate resources. MHB does not correlate the contents of the data for each single visit. For example, when a patient goes to see a doctor for a fever, the clinical information (medical institution details, medical date, disease diagnosis, personal payment amount, health insurance payment points, and prescription details) will be stored in the same section. However, the information between the same section is not connected. The next time the patient goes to see the doctor, a new section will be created. During FHIR-format data storage, each piece of data needs to be concatenated and referenced to the other pieces of data. In the future, data search and management efficiency can greatly improve.\n<\/p><p>This study provided FHIR-based personal health records and designed a FHIR-based personal health management platform. FHIR \u201cBundle\u201d is used to combine whole data into one document. FHIR Bundle is a resource that gathers a collection of different resources into a single document. We used it to bundle resources together as a single health record.\n<\/p><p>The structure of the conversion of the health passbook into the FHIR format in this study is shown in Figure 3. The FHIR structure uses a three-tier design for format conversion. The first layer (black) uses the Bundle resource to integrate all the information of the MHB. The second layer (blue) uses a total of 11 different resources to store the detailed information of the MHB according to the information contained in the health passbook. The third layer (red) is the index between each data. In FHIR, the data will be scattered everywhere and then linked to each other by index to ensure that the data is complete and easy to retrieve.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"ba5643552774c4339254c10f94e103b3\"><img alt=\"Fig3 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b3\/Fig3_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> FHIR format PHR data structure. PHR = personal health record.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>A composition resource is a set of healthcare-related information that can connect all single pieces of information for one medical service and has clinical attestation.\n<\/p><p>In this study, composition resources were used to connect records with different pieces of data. The composition resources included patient information (reference to patient resources), medical institution details (reference to organization resources), medical service information (reference to encounter resources), disease diagnoses (reference to condition resources), personal payment amounts (reference to coverage resources), health insurance payment points (reference to coverage resources), treatment details (reference to procedure resources), prescription details (reference to medication request resources), and testing reports (reference to observation resources).\n<\/p><p>The FHIR Encounter resource was used to record the interaction between patient and healthcare provider. The data include healthcare services records and the medical service place.\n<\/p><p>The Encounter resource was used to record the information of each medical service, including clinical diagnoses, observations, medical procedures, and immunizations (reference to different resources). Due to the different structures of MHB and the FHIR format, it cannot connect the medication information to the Encounter resource directly. In our design, the Encounter resource was connected to the Composition resource for connecting all information.\n<\/p><p>The PHR from a different country should be converted to the same standard for cross-country interoperability PHR implementation. FHIR Release #4 (version 4.0.1) was implemented in this study.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Personal_health_record_management_platform_design\">Personal health record management platform design<\/span><\/h3>\n<p>A platform was developed to view and transform PHRs. The platform contained two main modules. The architecture of the platform and module is shown in Figure 4. The MHB conversion module was used to convert the MHB data downloaded from the NHIA to the FHIR format. The index in the MHB was parsed and mapped to a separate FHIR resource. After data processing, an FHIR-format PHR could be created through this module. An HL7 application programming interface (HAPI) FHIR server was used as a health data server in our platform. HAPI FHIR is an open-source code that can fully implement the FHIR standard for interoperability in healthcare. It can store health data that comply with the FHIR format. Health data can be managed and transmitted based on a HAPI server, and HAPI can be used to verify the correctness of the FHIR format. If the format is not FHIR-compliant, the upload will be rejected.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"3a2d3ecfbe63fb7a94012b514947cbc0\"><img alt=\"Fig4 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d7\/Fig4_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Architecture of the platform and module. MHB = My Health Bank, RWD = real-world data, HAPI = Health Level Seven International Application Programming Interface.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The PHR viewer module could be used to view and manage personal health records, and PHR JSON files based on the FHIR format could be transferred by the viewer module.\n<\/p><p>For the design of data interoperability, the files after FHIR format conversion through the platform can be downloaded from the platform. The development of this research platform uses Windows 10 as the basic environment, and uses ASP.NET to set up a front visualization display platform and a back-end database data transmission and acceptance platform. The database uses a HAPI Server.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authentication_and_security\">Authentication and security<\/span><\/h3>\n<p>The architecture of authentication and security mechanism in this platform is shown in Figure 5. Network Time Protocol (NTP)\/Simple Network Time Protocol (SNTP) was used in the security and user authentication mechanism when processing and applying personal health information. The correct time was sourced from the National Time and Frequency Standard Laboratory. Transport Layer Security (TLS) 1.2 (such as HTTP Secure (HTTPS)) was used for protection in the data communication part. TLS 1.2 was released in 2008 and can operate on HTTP\/2. Currently, mainstream browsers only implement encrypted HTTP\/2, which makes HTTP\/2 + TLS 1.2 or later a mandatory standard.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"7aa3f300838d8131fd7094b2dea3ea64\"><img alt=\"Fig5 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/43\/Fig5_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Authentication and security mechanism in this platform. NTP = Network Time Protocol, SNTP = Simple Network Time Protocol, TLS = Transport Layer Security.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>User authentication is a critical part in this platform. Confirmation of legitimate user access can ensure data security. Referring to HL7 official recommendations, we used OAuth (OAuth 2.0) in the user authentication mechanism. OAuth2 is an authorization framework that allows users access to the user\u2019s private resources (such as photos, videos, and contact lists) stored on a certain website without a username and password through a third-party application. Third-party applications can obtain limited access to user accounts on OAuth2 providers (such as Microsoft, Google, Facebook, or Twitter) through HTTP\/HTTPS services.\n<\/p><p>This system does not verify the user itself (such as account\/password), but uses it in the OAuth2 environment where the OAuth2 provider is responsible for user authentication. Usually, when a user wants to log into the system, it will interactively redirect the user to the OAuth2 provider for authentication and obtain an OAuth2 token. Then, it sends back the HTTP request and provides the OAuth2 token in the HTTP Authentication header to represent the user. The system obtains the OAuth2 token and uses the OAuth2 authorization server to verify it and complete the login. In this way, system users can use accounts from Microsoft, Google, Facebook, or Twitter to log in with existing verification procedures, and only after logging into the system can they be associated with user data in the system.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Interoperability_PHR:_Profile_of_MHB_PHR_based_on_FHIR_format\">Interoperability PHR: Profile of MHB PHR based on FHIR format<\/span><\/h3>\n<p>The MHB PHRs based on FHIR format were implemented in this study. A total of 11 resources were included in our PHRs, including patients, composition, encounters, conditions, observations, procedures, medication requests, allergy intolerances, immunization recommendations, coverage, and organizations. The MHB data were converted to the FHIR format and stored in the personal health management platform.\n<\/p><p>The Bundle resource was used to connect the whole records. Different resources could be referenced through the subject element, which integrated the entire medical record of each medical service under the Composition resource. The Patient resource included personal information such as name, telephone number, gender, birthday, and address.\n<\/p><p>Encounter resource data referred to medical service information, including diagnoses and service providers, and could be referenced to Condition, Procedure, Observation, Condition, Patient, and Organization. Condition resource data referred to diagnoses and could be referenced to Encounter and Patient. Observation data referred to testing reports, including type of testing and value, and could be referenced to Organization, Patient, and Encounter. Procedure referred to treatment details and could be referenced to Patient, Encounter, and Condition. Medication request data referred to prescription details, including the drug, dosage, and code, and could be referenced to Patient and Encounter. Allergy intolerance data referred to allergy information and could be referenced to Patient. Coverage data referred to payment information, including personal payment amounts and health insurance payment points, and could be referenced to Patient and Encounter. Organization data referred to organization information, including name, telephone number, and address. The structure of different resource connections is shown in Figure 6.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"1ec65c0cf3bfcc0f4f7f835365b1740a\"><img alt=\"Fig6 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f4\/Fig6_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> Structure of different resource connections.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In Figure 6, each box represents a different resource. We converted the MHB single medical service record into a data format that conforms to the FHIR framework to meet international standards and subsequent data management applications.\n<\/p><p>The box representing each resource describes the relevant data that can be stored and the amount of data that can be stored. Take the composition resource on the left in Figure 5 as an example: it can contain different items of status, type, subject, encounter, date, author, title, and section. The field \u201c0..*\u201d means the amount of different data that can be stored. Take status as an example: the content of the field \u201c0..*\u201d is \u201c1..1\u201d, which means the status must have a data point and only can have one data point. Take section as another example: the content of the field \u201c0..*\u201d is \u201c0..*\u201d, which means that there can be no section data, or an unlimited number of sections (* means unlimited number).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_conversion_of_international_format\">Data conversion of international format<\/span><\/h3>\n<p>The conversion result of the MHB XML file into the FHIR standard of JSON format is shown in Figure 7. The different types of forms in the MHB were disassembled into different types of FHIR. According to the data format of each medical visit record in the MHB, the relevant data are referenced to the Composition resource. Each resource is also referenced to others.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"c9b9a71286a2efafe85cac5a733a73d9\"><img alt=\"Fig7 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/06\/Fig7_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 7.<\/b> The convert result of the personal health record.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For example, the Encounter resource references all information about patient medical records, including resources such as Patient, Condition, Observation, Procedure, and Organization. Therefore, the information of each record can be retrieved from the Encounter resource. However, according to the FHIR standard, Encounter cannot reference medication and medical orders, so the Composition resource is used to concatenate the medication and treatment orders of each record to achieve the correlation between the data.\n<\/p><p>The main core of the data conversion module is the use of custom filters and conversion template technology. The data conversion module architecture is shown in Figure 8. Since the source data format of MHB can be XML or JSON, XPath and JSPath technologies are applied to filter out specific information modules. Then, we combined the corresponding defined MHB and FHIR format mapping logic template for conversion. In the template, items\/attributes, coding, and even logic rules are described for the information modules of the source and target. Through this model, whether it is a new version or a new item of MHB or a new version of FHIR in the future, resources can be easily mapped, and FHIR data (XML\/JSON) conforming to the latest specifications can be successfully produced.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"6d1f728dc66ab02cc31fb5089fbaabea\"><img alt=\"Fig8 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/eb\/Fig8_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 8.<\/b> Data conversion module technology architecture.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Personal_health_management_platform\">Personal health management platform<\/span><\/h3>\n<p>After the user uploads the MHB data to the platform, the PHRs will be automatically converted to the FHIR format. PHR data can be displayed in a platform for personal health management. An example of a personal health record and user interface is shown in Figure 9. Record examples include patient information, diagnoses, service provider organization information, medication information, and coverage. Health records in the FHIR format can be displayed and stored on the platform, and users can download PHRs in FHIR format for record management or data exchange. The platform has achieved international interoperability goals, and data exchange efficiency and high quality can be improved through the platform.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig9_Lee_Sustain20_13-1.png\" class=\"image wiki-link\" data-key=\"6e7477fa6df3d7e9ad8fb8fa96fb55de\"><img alt=\"Fig9 Lee Sustain20 13-1.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/95\/Fig9_Lee_Sustain20_13-1.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 9.<\/b> An example of a personal health record and platform user interface.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Data_verification_mechanism\">Data verification mechanism<\/span><\/h3>\n<p>After the data are converted, the correctness of the data format needs to be verified. A HAPI server is used in the data verification mechanism.\n<\/p><p>HAPI is open-source software designed by a group of experts in the medical information field. Its architecture complies with FHIR standards and procedures. It has end-to-end code hygiene, secure defaults, regular updates, advanced features, reliability, predictable ownership, and in-house security features of expertise. After the HAPI server is established, data can be uploaded in an FHIR-compliant form.\n<\/p><p>In the data conversion process of this research, the converted FHIR JSON file was sent to the HAPI server to confirm that it is correct before the entire file was saved and output.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>The interoperability of the PHR format and international data exchange are essential in any kind of medical service. The interoperability of standards-based communication of system issue was previously analyzed by Urbauer <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-UrbauerAppl15_12-1\" class=\"reference\"><a href=\"#cite_note-UrbauerAppl15-12\">[12]<\/a><\/sup> They discussed the gaps in system interoperability communication and suggested possible solutions. Alyami <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-AlyamiRemov16_9-1\" class=\"reference\"><a href=\"#cite_note-AlyamiRemov16-9\">[9]<\/a><\/sup> and Roehrs <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-RoehrsPerson17_13-1\" class=\"reference\"><a href=\"#cite_note-RoehrsPerson17-13\">[13]<\/a><\/sup> proposed the key point of interoperability of personal health records. A service-oriented architecture for developing an interoperable integrated PHR system was presented by Li.<sup id=\"rdp-ebb-cite_ref-LiAServ17_11-1\" class=\"reference\"><a href=\"#cite_note-LiAServ17-11\">[11]<\/a><\/sup> And Roehrs <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-RoehrsToward19_27-1\" class=\"reference\"><a href=\"#cite_note-RoehrsToward19-27\">[27]<\/a><\/sup> proposed an application model for integrating different standards in PHRs to solve the interoperability problem.\n<\/p><p>Although many studies have discussed interoperability, it is inefficient to design a unique system for each different topic and goal. The HL7 FHIR achieved interoperability and system development efficiency by taking a different approach. Instead of using traditional data storage methods, HL7 FHIR uses a modular approach separating healthcare and medical service data (such as patient information, diagnoses, treatments, medications, and service records, etc.) into independent modules, which are called resources. The FHIR record control architecture is based on APIs and RESTful web services (the backbone of modern web applications) to achieve efficient and interoperable PHR control by utilizing modern web and mobile technology designs (such as HTTP-based lightweight REST protocol, JSON, or RDF, etc.).\n<\/p><p>According to the research of Saripalle <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SaripalleUsing19_37-0\" class=\"reference\"><a href=\"#cite_note-SaripalleUsing19-37\">[37]<\/a><\/sup>, FHIR can be used as the basic structure of PHR and has the ability to customize the PHR. At present, Taiwan does not have any data that meet international standards and regulations, which leads to a gap in international medical treatment. In the era of globalization, data should be integrated and applied as soon as possible to improve the convenience of data exchange, access, and use.\n<\/p><p>On the other hand, under the application of <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">data analysis<\/a> and big-data-related <a href=\"https:\/\/www.limswiki.org\/index.php\/Artificial_intelligence\" title=\"Artificial intelligence\" class=\"wiki-link\" data-key=\"0c45a597361ca47e1cd8112af676276e\">artificial intelligence<\/a> (AI), more and more attention is paid to data conforming to international standards, and only data conforming to standards can be quickly integrated and applied. Real-world evidence (RWE) research, for example, needs to ensure that key results of effectiveness can be observed and established in real-world data (RWD) sources, and a clinical RWD database can provide correct data to affect the overall research design and its clinical and cost burden. It may be necessary for a data repository to incorporate new data with the FHIR international format into the PHR or to collect and store biomarkers for subsequent analysis. In clinical research, studies such as the Salford Lung Study<sup id=\"rdp-ebb-cite_ref-BakerlyTheSalford15_38-0\" class=\"reference\"><a href=\"#cite_note-BakerlyTheSalford15-38\">[38]<\/a><\/sup> have proven that the use of RWD data can make potential progress in generating RWE, but it has also found many challenges related to infrastructure and research funding resources.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>Although many studies have made clinical data management systems that meet international standards, most of them are based on patients with specific diseases, and data interoperability is still inflexible. The innovation of this research is to take personal health records as the main body of the contents and apply international standards to establish a demonstration platform. Through the integration of different technologies\u2014such as Xpath and JSPath for data conversion, HAPI server for data verification, and OAuth for authentication\u2014an interoperable platform with personal health records as the main body has been constructed.\n<\/p><p>This study leveraged HL7 FHIR to achieve interoperability as well as capture and share data using FHIR resources and formats. Taiwan\u2019s My Health Bank personal health records, launched by the NHI, was used as an international prototype. Our platform can perform personal health control and data management and provide PHRs based on FHIR. Users can exchange PHRs easily and efficiency. Based on the FHIR format, the platform can not only provide PHRs in the FHIR format but also capture any healthcare data based on FHIR. The main purposes of the platform were to demonstrate the content of HL7 FHIR resources and personal health information (profiles, health records, and functional standards) and to design and implement interoperable PHRs that are compatible with HL7 PHRs. The HL7 FHIR was a specification and standard, and the HAPI FHIR, which is a Java implementation of HL7 FHIR, was used as the database. The data captured in the PHR were structured as an FHIR resource and shared with the EHR in the JSON format using web services. The conversion of data was verified by HAPI in the correct format, and it was provided for users to download and view on website services. In general, this study allowed PHRs to be interoperable and easily exchanged and an architecture for international data exchange to be proposed.\n<\/p><p>Comparing our study to other studies, the degree of interoperability is higher and the applicable population is larger (not confined by specific health problems). There are very few restrictions to adopting our platform since we use HL7 FHIR and all other open-source resources, and the personal health management can be implemented since the MHB provides most of the information we need to do health management. Our platform provides a user-friendly authentication process based on the OAuth mechanism. Therefore, in this study, we have completed an innovative application framework in Taiwan. The application we developed will be very useful for a single person to use or for other system developers to implement their own standard PHR software.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Conceptualization, Y.-L.L., H.-A.L., and C.-Y.H.; methodology, Y.-L.L. and H.-A.L.; software, H.-H.K.; validation, H.-A.L.; system structure design, Y.-L.L. and H.-A.L.; writing\u2014original draft preparation, Y.-L.L. and H.-A.L.; writing\u2014review and editing, C.-Y.H. and H.-W.C.; supervision, H.-W.C. All authors have read and agreed to the published version of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research was funded by Ministry of Science and Technology, Taiwan grant number 109-2221-E-227 -003 -MY2.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-TangPerson06-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TangPerson06_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tang, P.C.; Ash, J.S.; Bates, D.W. et al. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447551\" target=\"_blank\">\"Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption\"<\/a>. <i>JAMIA<\/i> <b>13<\/b> (2): 121\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2025\" target=\"_blank\">10.1197\/jamia.M2025<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1447551\/\" target=\"_blank\">PMC1447551<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16357345\" target=\"_blank\">16357345<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447551\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447551<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Personal+health+records%3A+definitions%2C+benefits%2C+and+strategies+for+overcoming+barriers+to+adoption&rft.jtitle=JAMIA&rft.aulast=Tang%2C+P.C.%3B+Ash%2C+J.S.%3B+Bates%2C+D.W.+et+al.&rft.au=Tang%2C+P.C.%3B+Ash%2C+J.S.%3B+Bates%2C+D.W.+et+al.&rft.date=2006&rft.volume=13&rft.issue=2&rft.pages=121%E2%80%936&rft_id=info:doi\/10.1197%2Fjamia.M2025&rft_id=info:pmc\/PMC1447551&rft_id=info:pmid\/16357345&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1447551&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarkleConnect03-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarkleConnect03_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Markle Foundation (01 July 2003). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.markle.org\/sites\/default\/files\/final_phwg_report1.pdf\" target=\"_blank\">\"Connecting for Health: A Public-Private Collaborative\"<\/a> (PDF). Markle Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.markle.org\/sites\/default\/files\/final_phwg_report1.pdf\" target=\"_blank\">https:\/\/www.markle.org\/sites\/default\/files\/final_phwg_report1.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 March 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Connecting+for+Health%3A+A+Public-Private+Collaborative&rft.atitle=&rft.aulast=Markle+Foundation&rft.au=Markle+Foundation&rft.date=01+July+2003&rft.pub=Markle+Foundation&rft_id=https%3A%2F%2Fwww.markle.org%2Fsites%2Fdefault%2Ffiles%2Ffinal_phwg_report1.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PowerActive17-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PowerActive17_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bower, J.K.; Bollinger, C.E.; Foraker, R.E. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5340503\" target=\"_blank\">\"Active Use of Electronic Health Records (EHRs) and Personal Health Records (PHRs) for Epidemiologic Research: Sample Representativeness and Nonresponse Bias in a Study of Women During Pregnancy\"<\/a>. <i>EGEMS<\/i> <b>5<\/b> (1): 1263. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13063%2F2327-9214.1263\" target=\"_blank\">10.13063\/2327-9214.1263<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5340503\/\" target=\"_blank\">PMC5340503<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28303255\" target=\"_blank\">28303255<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5340503\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5340503<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Active+Use+of+Electronic+Health+Records+%28EHRs%29+and+Personal+Health+Records+%28PHRs%29+for+Epidemiologic+Research%3A+Sample+Representativeness+and+Nonresponse+Bias+in+a+Study+of+Women+During+Pregnancy&rft.jtitle=EGEMS&rft.aulast=Bower%2C+J.K.%3B+Bollinger%2C+C.E.%3B+Foraker%2C+R.E.+et+al.&rft.au=Bower%2C+J.K.%3B+Bollinger%2C+C.E.%3B+Foraker%2C+R.E.+et+al.&rft.date=2017&rft.volume=5&rft.issue=1&rft.pages=1263&rft_id=info:doi\/10.13063%2F2327-9214.1263&rft_id=info:pmc\/PMC5340503&rft_id=info:pmid\/28303255&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5340503&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BonanderPublic10-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BonanderPublic10_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bonanader, J.; Gates, S. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2956336\" target=\"_blank\">\"Public health in an era of personal health records: Opportunities for innovation and new partnerships\"<\/a>. <i>Journal of Medical Internet Research<\/i> <b>12<\/b> (3): e33. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fjmir.1346\" target=\"_blank\">10.2196\/jmir.1346<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2956336\/\" target=\"_blank\">PMC2956336<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20699216\" target=\"_blank\">20699216<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2956336\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2956336<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Public+health+in+an+era+of+personal+health+records%3A+Opportunities+for+innovation+and+new+partnerships&rft.jtitle=Journal+of+Medical+Internet+Research&rft.aulast=Bonanader%2C+J.%3B+Gates%2C+S.&rft.au=Bonanader%2C+J.%3B+Gates%2C+S.&rft.date=2010&rft.volume=12&rft.issue=3&rft.pages=e33&rft_id=info:doi\/10.2196%2Fjmir.1346&rft_id=info:pmc\/PMC2956336&rft_id=info:pmid\/20699216&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2956336&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TimmermanWhat13-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TimmermanWhat13_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Timmerman, L. (04 February 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/xconomy.com\/national\/2013\/02\/04\/whats-in-a-name-a-lot-when-it-comes-to-precision-medicine\/\" target=\"_blank\">\"What\u2019s in a Name? A Lot, When It Comes to \u2018Precision Medicine\u2019\"<\/a>. <i>Xconomy<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/xconomy.com\/national\/2013\/02\/04\/whats-in-a-name-a-lot-when-it-comes-to-precision-medicine\/\" target=\"_blank\">https:\/\/xconomy.com\/national\/2013\/02\/04\/whats-in-a-name-a-lot-when-it-comes-to-precision-medicine\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What%E2%80%99s+in+a+Name%3F+A+Lot%2C+When+It+Comes+to+%E2%80%98Precision+Medicine%E2%80%99&rft.atitle=Xconomy&rft.aulast=Timmerman%2C+L.&rft.au=Timmerman%2C+L.&rft.date=04+February+2013&rft_id=https%3A%2F%2Fxconomy.com%2Fnational%2F2013%2F02%2F04%2Fwhats-in-a-name-a-lot-when-it-comes-to-precision-medicine%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IrelandReveal19-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IrelandReveal19_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Ireland, S. (05 August 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ceoworld.biz\/2019\/08\/05\/revealed-countries-with-the-best-health-care-systems-2019\/\" target=\"_blank\">\"Revealed: Countries With The Best Health Care Systems, 2019\"<\/a>. <i>CEOWORLD magazine<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ceoworld.biz\/2019\/08\/05\/revealed-countries-with-the-best-health-care-systems-2019\/\" target=\"_blank\">https:\/\/ceoworld.biz\/2019\/08\/05\/revealed-countries-with-the-best-health-care-systems-2019\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Revealed%3A+Countries+With+The+Best+Health+Care+Systems%2C+2019&rft.atitle=CEOWORLD+magazine&rft.aulast=Ireland%2C+S.&rft.au=Ireland%2C+S.&rft.date=05+August+2019&rft_id=https%3A%2F%2Fceoworld.biz%2F2019%2F08%2F05%2Frevealed-countries-with-the-best-health-care-systems-2019%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IqbalIsLong15-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IqbalIsLong15_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Iqbal, U.; Nguyen, P.-A.; Syed-Abdul, S. et al.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4602739\" target=\"_blank\"><i>Is long-term use of benzodiazepine a risk for cancer?<\/i><\/a>. <b>94<\/b>. e483. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FMD.0000000000000483\" target=\"_blank\">10.1097\/MD.0000000000000483<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4602739\/\" target=\"_blank\">PMC4602739<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25674736\" target=\"_blank\">25674736<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4602739\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4602739<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Is+long-term+use+of+benzodiazepine+a+risk+for+cancer%3F&rft.aulast=Iqbal%2C+U.%3B+Nguyen%2C+P.-A.%3B+Syed-Abdul%2C+S.+et+al.&rft.au=Iqbal%2C+U.%3B+Nguyen%2C+P.-A.%3B+Syed-Abdul%2C+S.+et+al.&rft.volume=94&rft.issue=6&rft.pages=e483&rft_id=info:doi\/10.1097%2FMD.0000000000000483&rft_id=info:pmc\/PMC4602739&rft_id=info:pmid\/25674736&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4602739&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JungSupport17-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JungSupport17_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jung, S.Y.; Lee, K.; Hwang, H. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5344982\" target=\"_blank\">\"Support for Sustainable Use of Personal Health Records: Understanding the Needs of Users as a First Step Towards Patient-Driven Mobile Health\"<\/a>. <i>JMIR mHealth and uHealth<\/i> <b>5<\/b> (2): e19. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fmhealth.6021\" target=\"_blank\">10.2196\/mhealth.6021<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5344982\/\" target=\"_blank\">PMC5344982<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28232300\" target=\"_blank\">28232300<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5344982\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5344982<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Support+for+Sustainable+Use+of+Personal+Health+Records%3A+Understanding+the+Needs+of+Users+as+a+First+Step+Towards+Patient-Driven+Mobile+Health&rft.jtitle=JMIR+mHealth+and+uHealth&rft.aulast=Jung%2C+S.Y.%3B+Lee%2C+K.%3B+Hwang%2C+H.+et+al.&rft.au=Jung%2C+S.Y.%3B+Lee%2C+K.%3B+Hwang%2C+H.+et+al.&rft.date=2017&rft.volume=5&rft.issue=2&rft.pages=e19&rft_id=info:doi\/10.2196%2Fmhealth.6021&rft_id=info:pmc\/PMC5344982&rft_id=info:pmid\/28232300&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5344982&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlyamiRemov16-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AlyamiRemov16_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-AlyamiRemov16_9-1\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alyami, M.A.; Song, Y.-T. (2016). \"Removing barriers in using personal health record systems\". <i>Proceedings of the 2016 IEEE\/ACIS 15th International Conference on Computer and Information Science<\/i>: 1\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FICIS.2016.7550810\" target=\"_blank\">10.1109\/ICIS.2016.7550810<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Removing+barriers+in+using+personal+health+record+systems&rft.jtitle=Proceedings+of+the+2016+IEEE%2FACIS+15th+International+Conference+on+Computer+and+Information+Science&rft.aulast=Alyami%2C+M.A.%3B+Song%2C+Y.-T.&rft.au=Alyami%2C+M.A.%3B+Song%2C+Y.-T.&rft.date=2016&rft.pages=1%E2%80%938&rft_id=info:doi\/10.1109%2FICIS.2016.7550810&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PlastirasCombin17-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PlastirasCombin17_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Plastiras, P.; O'Sullivan, D.M. (2017). \"Combining Ontologies and Open Standards to Derive a Middle Layer Information Model for Interoperability of Personal and Electronic Health Records\". <i>Journal of Medical Systems<\/i> <b>41<\/b> (12): 195. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10916-017-0838-9\" target=\"_blank\">10.1007\/s10916-017-0838-9<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29081012\" target=\"_blank\">29081012<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Combining+Ontologies+and+Open+Standards+to+Derive+a+Middle+Layer+Information+Model+for+Interoperability+of+Personal+and+Electronic+Health+Records&rft.jtitle=Journal+of+Medical+Systems&rft.aulast=Plastiras%2C+P.%3B+O%27Sullivan%2C+D.M.&rft.au=Plastiras%2C+P.%3B+O%27Sullivan%2C+D.M.&rft.date=2017&rft.volume=41&rft.issue=12&rft.pages=195&rft_id=info:doi\/10.1007%2Fs10916-017-0838-9&rft_id=info:pmid\/29081012&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiAServ17-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiAServ17_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiAServ17_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, J. (2017). \"A Service-Oriented Approach to Interoperable and Secure Personal Health Record Systems\". <i>Proceedings of the 2017 IEEE Symposium on Service-Oriented System Engineering<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FSOSE.2017.20\" target=\"_blank\">10.1109\/SOSE.2017.20<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Service-Oriented+Approach+to+Interoperable+and+Secure+Personal+Health+Record+Systems&rft.jtitle=Proceedings+of+the+2017+IEEE+Symposium+on+Service-Oriented+System+Engineering&rft.aulast=Li%2C+J.&rft.au=Li%2C+J.&rft.date=2017&rft_id=info:doi\/10.1109%2FSOSE.2017.20&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UrbauerAppl15-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-UrbauerAppl15_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-UrbauerAppl15_12-1\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Urbauer, P.; Sauermann, S.; Frohner, M. et al. (2015). \"Applicability of IHE\/Continua components for PHR systems: Learning from experiences\". <i>Computers in Biology and Medicine<\/i> <b>59<\/b>: 186\u201393. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.compbiomed.2013.12.003\" target=\"_blank\">10.1016\/j.compbiomed.2013.12.003<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24374230\" target=\"_blank\">24374230<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Applicability+of+IHE%2FContinua+components+for+PHR+systems%3A+Learning+from+experiences&rft.jtitle=Computers+in+Biology+and+Medicine&rft.aulast=Urbauer%2C+P.%3B+Sauermann%2C+S.%3B+Frohner%2C+M.+et+al.&rft.au=Urbauer%2C+P.%3B+Sauermann%2C+S.%3B+Frohner%2C+M.+et+al.&rft.date=2015&rft.volume=59&rft.pages=186%E2%80%9393&rft_id=info:doi\/10.1016%2Fj.compbiomed.2013.12.003&rft_id=info:pmid\/24374230&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoehrsPerson17-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RoehrsPerson17_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-RoehrsPerson17_13-1\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Roehrs, A.; da Costa, C.A.; da Rosa Righi, R. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5251169\" target=\"_blank\">\"Personal Health Records: A Systematic Literature Review\"<\/a>. <i>Journal of Medical Internet Research<\/i> <b>19<\/b> (1): e13. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fjmir.5876\" target=\"_blank\">10.2196\/jmir.5876<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5251169\/\" target=\"_blank\">PMC5251169<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28062391\" target=\"_blank\">28062391<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5251169\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5251169<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Personal+Health+Records%3A+A+Systematic+Literature+Review&rft.jtitle=Journal+of+Medical+Internet+Research&rft.aulast=Roehrs%2C+A.%3B+da+Costa%2C+C.A.%3B+da+Rosa+Righi%2C+R.+et+al.&rft.au=Roehrs%2C+A.%3B+da+Costa%2C+C.A.%3B+da+Rosa+Righi%2C+R.+et+al.&rft.date=2017&rft.volume=19&rft.issue=1&rft.pages=e13&rft_id=info:doi\/10.2196%2Fjmir.5876&rft_id=info:pmc\/PMC5251169&rft_id=info:pmid\/28062391&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5251169&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HL7_FHIR-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HL7_FHIR_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/hl7.org\/FHIR\/\" target=\"_blank\">\"Welcome to FHIR\"<\/a>. Health Level 7. 01 November 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/hl7.org\/FHIR\/\" target=\"_blank\">https:\/\/hl7.org\/FHIR\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 26 April 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Welcome+to+FHIR&rft.atitle=&rft.date=01+November+2019&rft.pub=Health+Level+7&rft_id=https%3A%2F%2Fhl7.org%2FFHIR%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchleyerPrelim19-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchleyerPrelim19_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schleyer, T.K.L.; Rahurkar, S.; Baublet, A.M. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6568135\" target=\"_blank\">\"Preliminary evaluation of the <i>Chest Pain Dashboard<\/i>, a FHIR-based approach for integrating health information exchange information directly into the clinical workflow\"<\/a>. <i>AMIA Joint Summits on Translational Science<\/i> <b>2019<\/b>: 656-664. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6568135\/\" target=\"_blank\">PMC6568135<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31259021\" target=\"_blank\">31259021<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6568135\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6568135<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Preliminary+evaluation+of+the+%27%27Chest+Pain+Dashboard%27%27%2C+a+FHIR-based+approach+for+integrating+health+information+exchange+information+directly+into+the+clinical+workflow&rft.jtitle=AMIA+Joint+Summits+on+Translational+Science&rft.aulast=Schleyer%2C+T.K.L.%3B+Rahurkar%2C+S.%3B+Baublet%2C+A.M.+et+al.&rft.au=Schleyer%2C+T.K.L.%3B+Rahurkar%2C+S.%3B+Baublet%2C+A.M.+et+al.&rft.date=2019&rft.volume=2019&rft.pages=656-664&rft_id=info:pmc\/PMC6568135&rft_id=info:pmid\/31259021&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6568135&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoodmanHealthy18-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GoodmanHealthy18_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goodman, A.B.; Braun, P.; Braunstein, M. (2018). \"Healthy Weight on FHIR - Innovative Technology to Support High Quality Clinical Care & Clinical to Community Linkages for Child Obesity\". <i>PEdiatrics<\/i> <b>141<\/b> (1 MeetingAbstract): 12. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1542%2Fpeds.141.1_MeetingAbstract.12\" target=\"_blank\">10.1542\/peds.141.1_MeetingAbstract.12<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Healthy+Weight+on+FHIR+-+Innovative+Technology+to+Support+High+Quality+Clinical+Care+%26+Clinical+to+Community+Linkages+for+Child+Obesity&rft.jtitle=PEdiatrics&rft.aulast=Goodman%2C+A.B.%3B+Braun%2C+P.%3B+Braunstein%2C+M.&rft.au=Goodman%2C+A.B.%3B+Braun%2C+P.%3B+Braunstein%2C+M.&rft.date=2018&rft.volume=141&rft.issue=1+MeetingAbstract&rft.pages=12&rft_id=info:doi\/10.1542%2Fpeds.141.1_MeetingAbstract.12&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaihanABlue18-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BaihanABlue18_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Baihan, M.S.; S\u00e1nchez, Y.K.R.; Shao, X. et al. (2018). \"Chapter 6: A Blueprint for Designing and Developing M-Health Applications for Diverse Stakeholders Utilizing FHIR\". In Rajkumar, R.. <i>Contemporary Applications of Mobile Computing in Healthcare Settings<\/i>. IGI Global. pp. 85\u2013124. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4018%2F978-1-5225-5036-5.ch006\" target=\"_blank\">10.4018\/978-1-5225-5036-5.ch006<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781522550365.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+6%3A+A+Blueprint+for+Designing+and+Developing+M-Health+Applications+for+Diverse+Stakeholders+Utilizing+FHIR&rft.atitle=Contemporary+Applications+of+Mobile+Computing+in+Healthcare+Settings&rft.aulast=Baihan%2C+M.S.%3B+S%C3%A1nchez%2C+Y.K.R.%3B+Shao%2C+X.+et+al.&rft.au=Baihan%2C+M.S.%3B+S%C3%A1nchez%2C+Y.K.R.%3B+Shao%2C+X.+et+al.&rft.date=2018&rft.pages=pp.%26nbsp%3B85%E2%80%93124&rft.pub=IGI+Global&rft_id=info:doi\/10.4018%2F978-1-5225-5036-5.ch006&rft.isbn=9781522550365&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DaumkeClinical19-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DaumkeClinical19_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Daumke, P.; Heitmann, K.U.; Heckmann, S. et al. (2019). \"Clinical Text Mining on FHIR\". <i>Studies in Health Technology and Informatics<\/i> <b>264<\/b>: 83\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2FSHTI190188\" target=\"_blank\">10.3233\/SHTI190188<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31437890\" target=\"_blank\">31437890<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Clinical+Text+Mining+on+FHIR&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Daumke%2C+P.%3B+Heitmann%2C+K.U.%3B+Heckmann%2C+S.+et+al.&rft.au=Daumke%2C+P.%3B+Heitmann%2C+K.U.%3B+Heckmann%2C+S.+et+al.&rft.date=2019&rft.volume=264&rft.pages=83%E2%80%937&rft_id=info:doi\/10.3233%2FSHTI190188&rft_id=info:pmid\/31437890&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StorckInter19-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StorckInter19_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Storck, M.; Hollenberg, L.; Dugas, M. et al. (2019). \"Interoperability Improvement of Mobile Patient Survey (MoPat) Implementing Fast Health Interoperability Resources (FHIR)\". <i>Studies in Health Technology and Informatics<\/i> <b>258<\/b>: 141\u201345. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30942732\" target=\"_blank\">30942732<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interoperability+Improvement+of+Mobile+Patient+Survey+%28MoPat%29+Implementing+Fast+Health+Interoperability+Resources+%28FHIR%29&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Storck%2C+M.%3B+Hollenberg%2C+L.%3B+Dugas%2C+M.+et+al.&rft.au=Storck%2C+M.%3B+Hollenberg%2C+L.%3B+Dugas%2C+M.+et+al.&rft.date=2019&rft.volume=258&rft.pages=141%E2%80%9345&rft_id=info:pmid\/30942732&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GiordanengoAFHIR18-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GiordanengoAFHIR18_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Giordanengo, A.; Bradway, M.; Gr\u00f8ttland, A. et al. (29 August 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ehealthresearch.no\/files\/documents\/Postere\/Poster_2018-02_16-9-ATTD-FHIR-based-Data-Flow.pdf\" target=\"_blank\">\"A FHIR-based Data Flow Enabling Patients with Diabetes to Share Self-collected Data with the Norwegian National Healthcare Systems and Electronic Health Records Systems\"<\/a> (PDF). <i>2017 Scandinavian Conference on Health Informatics<\/i>. Norwegian Centre for E-health Research<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ehealthresearch.no\/files\/documents\/Postere\/Poster_2018-02_16-9-ATTD-FHIR-based-Data-Flow.pdf\" target=\"_blank\">https:\/\/ehealthresearch.no\/files\/documents\/Postere\/Poster_2018-02_16-9-ATTD-FHIR-based-Data-Flow.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+FHIR-based+Data+Flow+Enabling+Patients+with+Diabetes+to+Share+Self-collected+Data+with+the+Norwegian+National+Healthcare+Systems+and+Electronic+Health+Records+Systems&rft.atitle=2017+Scandinavian+Conference+on+Health+Informatics&rft.aulast=Giordanengo%2C+A.%3B+Bradway%2C+M.%3B+Gr%C3%B8ttland%2C+A.+et+al.&rft.au=Giordanengo%2C+A.%3B+Bradway%2C+M.%3B+Gr%C3%B8ttland%2C+A.+et+al.&rft.date=29+August+2017&rft.pub=Norwegian+Centre+for+E-health+Research&rft_id=https%3A%2F%2Fehealthresearch.no%2Ffiles%2Fdocuments%2FPostere%2FPoster_2018-02_16-9-ATTD-FHIR-based-Data-Flow.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KiourtisStruct19-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KiourtisStruct19_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kiourtis, A.; Mavrogiorgou, A.; Menychtas, A. et al. (2019). \"Structurally Mapping Healthcare Data to HL7 FHIR through Ontology Alignment\". <i>Journal of Medical Systems<\/i> <b>43<\/b> (3): 62. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10916-019-1183-y\" target=\"_blank\">10.1007\/s10916-019-1183-y<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30721349\" target=\"_blank\">30721349<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Structurally+Mapping+Healthcare+Data+to+HL7+FHIR+through+Ontology+Alignment&rft.jtitle=Journal+of+Medical+Systems&rft.aulast=Kiourtis%2C+A.%3B+Mavrogiorgou%2C+A.%3B+Menychtas%2C+A.+et+al.&rft.au=Kiourtis%2C+A.%3B+Mavrogiorgou%2C+A.%3B+Menychtas%2C+A.+et+al.&rft.date=2019&rft.volume=43&rft.issue=3&rft.pages=62&rft_id=info:doi\/10.1007%2Fs10916-019-1183-y&rft_id=info:pmid\/30721349&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-G.C3.B8egAFutur18-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-G.C3.B8egAFutur18_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">G\u00f8eg, K.R.; Rasmussen, R.K.; Jensen, L. et al. (2018). \"A future-proof architecture for telemedicine using loose-coupled modules and HL7 FHIR\". <i>Computer Methods and Programs in Biomedicine<\/i> <b>160<\/b>: 95\u2013101. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cmpb.2018.03.010\" target=\"_blank\">10.1016\/j.cmpb.2018.03.010<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29728251\" target=\"_blank\">29728251<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+future-proof+architecture+for+telemedicine+using+loose-coupled+modules+and+HL7+FHIR&rft.jtitle=Computer+Methods+and+Programs+in+Biomedicine&rft.aulast=G%C3%B8eg%2C+K.R.%3B+Rasmussen%2C+R.K.%3B+Jensen%2C+L.+et+al.&rft.au=G%C3%B8eg%2C+K.R.%3B+Rasmussen%2C+R.K.%3B+Jensen%2C+L.+et+al.&rft.date=2018&rft.volume=160&rft.pages=95%E2%80%93101&rft_id=info:doi\/10.1016%2Fj.cmpb.2018.03.010&rft_id=info:pmid\/29728251&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlperAchiev19-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlperAchiev19_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alper, B.; Mayer, M.; Shahin, K. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ebm.bmj.com\/content\/24\/Suppl_1\/A15.1\" target=\"_blank\">\"Achieving evidence interoperability in the computer age: Setting evidence on FHIR\"<\/a>. <i>BMJ Evidence-Based Medicine<\/i> <b>24<\/b> (Suppl. 1): A15<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ebm.bmj.com\/content\/24\/Suppl_1\/A15.1\" target=\"_blank\">https:\/\/ebm.bmj.com\/content\/24\/Suppl_1\/A15.1<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Achieving+evidence+interoperability+in+the+computer+age%3A+Setting+evidence+on+FHIR&rft.jtitle=BMJ+Evidence-Based+Medicine&rft.aulast=Alper%2C+B.%3B+Mayer%2C+M.%3B+Shahin%2C+K.+et+al.&rft.au=Alper%2C+B.%3B+Mayer%2C+M.%3B+Shahin%2C+K.+et+al.&rft.date=2019&rft.volume=24&rft.issue=Suppl.+1&rft.pages=A15&rft_id=https%3A%2F%2Febm.bmj.com%2Fcontent%2F24%2FSuppl_1%2FA15.1&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LackerbauerAModel18-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LackerbauerAModel18_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lackerbauer, A.M.; Lin, A.C.; Krauss, O. et al. (2018). \"A Model for Implementing an Interoperable Electronic Consent Form for Medical Treatment Using HL7 FHIR\". <i>European Journal for Biomedical Informatics<\/i> <b>14<\/b> (3): 37\u201347. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.24105%2Fejbi.2018.14.3.6\" target=\"_blank\">10.24105\/ejbi.2018.14.3.6<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Model+for+Implementing+an+Interoperable+Electronic+Consent+Form+for+Medical+Treatment+Using+HL7+FHIR&rft.jtitle=European+Journal+for+Biomedical+Informatics&rft.aulast=Lackerbauer%2C+A.M.%3B+Lin%2C+A.C.%3B+Krauss%2C+O.+et+al.&rft.au=Lackerbauer%2C+A.M.%3B+Lin%2C+A.C.%3B+Krauss%2C+O.+et+al.&rft.date=2018&rft.volume=14&rft.issue=3&rft.pages=37%E2%80%9347&rft_id=info:doi\/10.24105%2Fejbi.2018.14.3.6&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DixonInteg19-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DixonInteg19_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dixon, B.E.; Taylor, D.E.; Choi, M. et al. (2019). \"Integration of FHIR to Facilitate Electronic Case Reporting: Results from a Pilot Study\". <i>Studies in Health Technology and Informatics<\/i> <b>264<\/b>: 940\u201344. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2FSHTI190362\" target=\"_blank\">10.3233\/SHTI190362<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31438062\" target=\"_blank\">31438062<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integration+of+FHIR+to+Facilitate+Electronic+Case+Reporting%3A+Results+from+a+Pilot+Study&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Dixon%2C+B.E.%3B+Taylor%2C+D.E.%3B+Choi%2C+M.+et+al.&rft.au=Dixon%2C+B.E.%3B+Taylor%2C+D.E.%3B+Choi%2C+M.+et+al.&rft.date=2019&rft.volume=264&rft.pages=940%E2%80%9344&rft_id=info:doi\/10.3233%2FSHTI190362&rft_id=info:pmid\/31438062&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HussainLearn18-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HussainLearn18_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hussain, M.A.; Langer, S.G.; Kohli, M. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5959839\" target=\"_blank\">\"Learning HL7 FHIR Using the HAPI FHIR Server and Its Use in Medical Imaging with the SIIM Dataset\"<\/a>. <i>Journal of Digital Imaging<\/i> <b>31<\/b> (3): 334\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-018-0090-y\" target=\"_blank\">10.1007\/s10278-018-0090-y<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5959839\/\" target=\"_blank\">PMC5959839<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29725959\" target=\"_blank\">29725959<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5959839\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5959839<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Learning+HL7+FHIR+Using+the+HAPI+FHIR+Server+and+Its+Use+in+Medical+Imaging+with+the+SIIM+Dataset&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=Hussain%2C+M.A.%3B+Langer%2C+S.G.%3B+Kohli%2C+M.+et+al.&rft.au=Hussain%2C+M.A.%3B+Langer%2C+S.G.%3B+Kohli%2C+M.+et+al.&rft.date=2018&rft.volume=31&rft.issue=3&rft.pages=334%E2%80%9340&rft_id=info:doi\/10.1007%2Fs10278-018-0090-y&rft_id=info:pmc\/PMC5959839&rft_id=info:pmid\/29725959&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5959839&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoehrsToward19-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RoehrsToward19_27-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-RoehrsToward19_27-1\">27.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Roehrs, A.; da Costa, C.A.; da Rosa Righi, R. et al. (2019). \"Toward a Model for Personal Health Record Interoperability\". <i>IEEE Journal of Biomedical and Health Informatics<\/i> <b>23<\/b> (2): 867\u201373. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FJBHI.2018.2836138\" target=\"_blank\">10.1109\/JBHI.2018.2836138<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29993759\" target=\"_blank\">29993759<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+Model+for+Personal+Health+Record+Interoperability&rft.jtitle=IEEE+Journal+of+Biomedical+and+Health+Informatics&rft.aulast=Roehrs%2C+A.%3B+da+Costa%2C+C.A.%3B+da+Rosa+Righi%2C+R.+et+al.&rft.au=Roehrs%2C+A.%3B+da+Costa%2C+C.A.%3B+da+Rosa+Righi%2C+R.+et+al.&rft.date=2019&rft.volume=23&rft.issue=2&rft.pages=867%E2%80%9373&rft_id=info:doi\/10.1109%2FJBHI.2018.2836138&rft_id=info:pmid\/29993759&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mart.C3.ADnez-Garc.C3.ADaWork15-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Mart.C3.ADnez-Garc.C3.ADaWork15_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mart\u00ednez-Garc\u00eda, A.; Garc\u00eda-Garc\u00eda, J.A.; Escalona, M.J. et al. (2015). \"Working with the HL7 metamodel in a Model Driven Engineering context\". <i>Journal of Biomedical Informatics<\/i> <b>57<\/b>: 415\u201324. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2015.09.001\" target=\"_blank\">10.1016\/j.jbi.2015.09.001<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26348545\" target=\"_blank\">26348545<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Working+with+the+HL7+metamodel+in+a+Model+Driven+Engineering+context&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Mart%C3%ADnez-Garc%C3%ADa%2C+A.%3B+Garc%C3%ADa-Garc%C3%ADa%2C+J.A.%3B+Escalona%2C+M.J.+et+al.&rft.au=Mart%C3%ADnez-Garc%C3%ADa%2C+A.%3B+Garc%C3%ADa-Garc%C3%ADa%2C+J.A.%3B+Escalona%2C+M.J.+et+al.&rft.date=2015&rft.volume=57&rft.pages=415%E2%80%9324&rft_id=info:doi\/10.1016%2Fj.jbi.2015.09.001&rft_id=info:pmid\/26348545&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WalderhaugExper08-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WalderhaugExper08_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Walderhaug, S.; Stav, E.; Mikalsen, M. (2008). \"Experiences from Model-Driven Development of Homecare Services: UML Profiles and Domain Models\". <i>Proceedings of MODELS 2008: International Conference on Model Driven Engineering Languages and Systems<\/i>: 199\u2013212. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-01648-6_22\" target=\"_blank\">10.1007\/978-3-642-01648-6_22<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Experiences+from+Model-Driven+Development+of+Homecare+Services%3A+UML+Profiles+and+Domain+Models&rft.jtitle=Proceedings+of+MODELS+2008%3A+International+Conference+on+Model+Driven+Engineering+Languages+and+Systems&rft.aulast=Walderhaug%2C+S.%3B+Stav%2C+E.%3B+Mikalsen%2C+M.&rft.au=Walderhaug%2C+S.%3B+Stav%2C+E.%3B+Mikalsen%2C+M.&rft.date=2008&rft.pages=199%E2%80%93212&rft_id=info:doi\/10.1007%2F978-3-642-01648-6_22&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Gonz.C3.A1lez-FerrerData12-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Gonz.C3.A1lez-FerrerData12_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Gonz\u00e1lez-Ferrer, A.; Peleg, M.; Verhees, B. et al. (2012). \"Data integration for clinical decision support based on open EHR archetypes and HL7 virtual medical record\". In Lenz, R.; Miksch, S.; Peleg, M. et al.. <i>Process Support and Knowledge Representation in Health Care<\/i>. Springer. pp. 71\u201384. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-36438-9\" target=\"_blank\">10.1007\/978-3-642-36438-9<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783642364389.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+integration+for+clinical+decision+support+based+on+open+EHR+archetypes+and+HL7+virtual+medical+record&rft.atitle=Process+Support+and+Knowledge+Representation+in+Health+Care&rft.aulast=Gonz%C3%A1lez-Ferrer%2C+A.%3B+Peleg%2C+M.%3B+Verhees%2C+B.+et+al.&rft.au=Gonz%C3%A1lez-Ferrer%2C+A.%3B+Peleg%2C+M.%3B+Verhees%2C+B.+et+al.&rft.date=2012&rft.pages=pp.%26nbsp%3B71%E2%80%9384&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-642-36438-9&rft.isbn=9783642364389&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SimonAUbiq13-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SimonAUbiq13_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Simon, S.K.; Anbananthen, K.S.M.; Lee, S. (2013). \"A Ubiquitous Personal Health Record (uPHR) Framework\". <i>Proceedings of the 2013 International Conference on Advanced Computer Science and Electronics Information<\/i>: 423\u201327. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2991%2Ficacsei.2013.105\" target=\"_blank\">10.2991\/icacsei.2013.105<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Ubiquitous+Personal+Health+Record+%28uPHR%29+Framework&rft.jtitle=Proceedings+of+the+2013+International+Conference+on+Advanced+Computer+Science+and+Electronics+Information&rft.aulast=Simon%2C+S.K.%3B+Anbananthen%2C+K.S.M.%3B+Lee%2C+S.&rft.au=Simon%2C+S.K.%3B+Anbananthen%2C+K.S.M.%3B+Lee%2C+S.&rft.date=2013&rft.pages=423%E2%80%9327&rft_id=info:doi\/10.2991%2Ficacsei.2013.105&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarcegliaAStand15-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarcegliaAStand15_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Marceglia, S.; Fontelo, P.; Rossi, E. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4586338\" target=\"_blank\">\"A Standards-Based Architecture Proposal for Integrating Patient mHealth Apps to Electronic Health Record Systems\"<\/a>. <i>Applied Clinical Informatics<\/i> <b>6<\/b> (3): 488-505. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4338%2FACI-2014-12-RA-0115\" target=\"_blank\">10.4338\/ACI-2014-12-RA-0115<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4586338\/\" target=\"_blank\">PMC4586338<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26448794\" target=\"_blank\">26448794<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4586338\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4586338<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Standards-Based+Architecture+Proposal+for+Integrating+Patient+mHealth+Apps+to+Electronic+Health+Record+Systems&rft.jtitle=Applied+Clinical+Informatics&rft.aulast=Marceglia%2C+S.%3B+Fontelo%2C+P.%3B+Rossi%2C+E.+et+al.&rft.au=Marceglia%2C+S.%3B+Fontelo%2C+P.%3B+Rossi%2C+E.+et+al.&rft.date=2015&rft.volume=6&rft.issue=3&rft.pages=488-505&rft_id=info:doi\/10.4338%2FACI-2014-12-RA-0115&rft_id=info:pmc\/PMC4586338&rft_id=info:pmid\/26448794&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4586338&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FondaEvol10-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FondaEvol10_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fonda, S.J.; Kedziora, R.J.; Vigersky, R.A. et al. (2010). \"Evolution of a web-based, prototype Personal Health Application for diabetes self-management\". <i>Journal of Biomedical Informatics<\/i> <b>43<\/b> (5 Suppl.): S17\u201321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2010.05.006\" target=\"_blank\">10.1016\/j.jbi.2010.05.006<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20937479\" target=\"_blank\">20937479<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evolution+of+a+web-based%2C+prototype+Personal+Health+Application+for+diabetes+self-management&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Fonda%2C+S.J.%3B+Kedziora%2C+R.J.%3B+Vigersky%2C+R.A.+et+al.&rft.au=Fonda%2C+S.J.%3B+Kedziora%2C+R.J.%3B+Vigersky%2C+R.A.+et+al.&rft.date=2010&rft.volume=43&rft.issue=5+Suppl.&rft.pages=S17%E2%80%9321&rft_id=info:doi\/10.1016%2Fj.jbi.2010.05.006&rft_id=info:pmid\/20937479&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BackonjaObserv12-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BackonjaObserv12_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Backonja, U.; Kim, K.; Casper, G.R. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3799183\" target=\"_blank\">\"Observations of daily living: putting the \"personal\" in personal health records\"<\/a>. <i>Proceedings of the 11th International Congress on Nursing Informatics<\/i> <b>2012<\/b>: 6. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3799183\/\" target=\"_blank\">PMC3799183<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24199037\" target=\"_blank\">24199037<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3799183\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3799183<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Observations+of+daily+living%3A+putting+the+%22personal%22+in+personal+health+records&rft.jtitle=Proceedings+of+the+11th+International+Congress+on+Nursing+Informatics&rft.aulast=Backonja%2C+U.%3B+Kim%2C+K.%3B+Casper%2C+G.R.+et+al.&rft.au=Backonja%2C+U.%3B+Kim%2C+K.%3B+Casper%2C+G.R.+et+al.&rft.date=2012&rft.volume=2012&rft.pages=6&rft_id=info:pmc\/PMC3799183&rft_id=info:pmid\/24199037&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3799183&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FergusonTowards10-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FergusonTowards10_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ferguson, G.; Quinn, J.; Horwitz, C. et al. (2010). \"Towards a Personal Health Management Assistant\". <i>Journal of Biomedical Informatics<\/i> <b>43<\/b> (5 Suppl.): S13\u201316. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2010.05.007\" target=\"_blank\">10.1016\/j.jbi.2010.05.007<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20937478\" target=\"_blank\">20937478<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+a+Personal+Health+Management+Assistant&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Ferguson%2C+G.%3B+Quinn%2C+J.%3B+Horwitz%2C+C.+et+al.&rft.au=Ferguson%2C+G.%3B+Quinn%2C+J.%3B+Horwitz%2C+C.+et+al.&rft.date=2010&rft.volume=43&rft.issue=5+Suppl.&rft.pages=S13%E2%80%9316&rft_id=info:doi\/10.1016%2Fj.jbi.2010.05.007&rft_id=info:pmid\/20937478&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SiekColor10-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SiekColor10_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Siek, K.A.; Ross, S.E.; Khan, D.U. et al. (2010). \"Colorado Care Tablet: The design of an interoperable Personal Health Application to help older adults with multimorbidity manage their medications\". <i>Journal of Biomedical Informatics<\/i> <b>43<\/b> (5 Suppl.): S22\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2010.05.014\" target=\"_blank\">10.1016\/j.jbi.2010.05.014<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20937480\" target=\"_blank\">20937480<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Colorado+Care+Tablet%3A+The+design+of+an+interoperable+Personal+Health+Application+to+help+older+adults+with+multimorbidity+manage+their+medications&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Siek%2C+K.A.%3B+Ross%2C+S.E.%3B+Khan%2C+D.U.+et+al.&rft.au=Siek%2C+K.A.%3B+Ross%2C+S.E.%3B+Khan%2C+D.U.+et+al.&rft.date=2010&rft.volume=43&rft.issue=5+Suppl.&rft.pages=S22%E2%80%936&rft_id=info:doi\/10.1016%2Fj.jbi.2010.05.014&rft_id=info:pmid\/20937480&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SaripalleUsing19-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SaripalleUsing19_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Saripalle, R.; Runyan, C.; Russell, M. (2019). \"Using HL7 FHIR to achieve interoperability in patient health record\". <i>Journal of Biomedical Informatics<\/i> <b>94<\/b>: 103188. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2019.103188\" target=\"_blank\">10.1016\/j.jbi.2019.103188<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31063828\" target=\"_blank\">31063828<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+HL7+FHIR+to+achieve+interoperability+in+patient+health+record&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Saripalle%2C+R.%3B+Runyan%2C+C.%3B+Russell%2C+M.&rft.au=Saripalle%2C+R.%3B+Runyan%2C+C.%3B+Russell%2C+M.&rft.date=2019&rft.volume=94&rft.pages=103188&rft_id=info:doi\/10.1016%2Fj.jbi.2019.103188&rft_id=info:pmid\/31063828&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BakerlyTheSalford15-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BakerlyTheSalford15_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bakerly, N.D.; Woodcock, A.; New, J.P. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4558879\" target=\"_blank\">\"The Salford Lung Study protocol: A pragmatic, randomised phase III real-world effectiveness trial in chronic obstructive pulmonary disease\"<\/a>. <i>Respiratory Research<\/i> <b>16<\/b> (1): 101. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12931-015-0267-6\" target=\"_blank\">10.1186\/s12931-015-0267-6<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4558879\/\" target=\"_blank\">PMC4558879<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26337978\" target=\"_blank\">26337978<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4558879\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4558879<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Salford+Lung+Study+protocol%3A+A+pragmatic%2C+randomised+phase+III+real-world+effectiveness+trial+in+chronic+obstructive+pulmonary+disease&rft.jtitle=Respiratory+Research&rft.aulast=Bakerly%2C+N.D.%3B+Woodcock%2C+A.%3B+New%2C+J.P.+et+al.&rft.au=Bakerly%2C+N.D.%3B+Woodcock%2C+A.%3B+New%2C+J.P.+et+al.&rft.date=2015&rft.volume=16&rft.issue=1&rft.pages=101&rft_id=info:doi\/10.1186%2Fs12931-015-0267-6&rft_id=info:pmc\/PMC4558879&rft_id=info:pmid\/26337978&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4558879&rfr_id=info:sid\/en.wikipedia.org:Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original reference 12 (Crabtree <i>et al.<\/i>) seems to have no bearing on the text and appears to have been accidentally included; it was omitted for this version. The same conclusion was reached about the original reference 38, which has also been omitted.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194101\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.904 seconds\nReal time usage: 3.591 seconds\nPreprocessor visited node count: 30193\/1000000\nPreprocessor generated node count: 38497\/1000000\nPost\u2010expand include size: 255459\/2097152 bytes\nTemplate argument size: 89920\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 614.494 1 - -total\n 88.75% 545.387 1 - Template:Reflist\n 75.39% 463.285 38 - Template:Citation\/core\n 65.96% 405.324 31 - Template:Cite_journal\n 9.47% 58.200 66 - Template:Citation\/identifier\n 8.40% 51.599 5 - Template:Cite_web\n 6.03% 37.036 1 - Template:Infobox_journal_article\n 5.75% 35.328 1 - Template:Infobox\n 4.64% 28.541 2 - Template:Cite_book\n 4.35% 26.741 43 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12409-0!*!0!!en!5!* and timestamp 20210429194057 and revision id 41958\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application\">https:\/\/www.limswiki.org\/index.php\/Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","bd8eee413799c826d26140a4bd9d594e_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b3\/Fig1_Lee_Sustain20_13-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/77\/Fig2_Lee_Sustain20_13-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b3\/Fig3_Lee_Sustain20_13-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d7\/Fig4_Lee_Sustain20_13-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/43\/Fig5_Lee_Sustain20_13-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f4\/Fig6_Lee_Sustain20_13-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/06\/Fig7_Lee_Sustain20_13-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/eb\/Fig8_Lee_Sustain20_13-1.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/95\/Fig9_Lee_Sustain20_13-1.png"],"bd8eee413799c826d26140a4bd9d594e_timestamp":1619725257,"f48cdd8c7fc25a130c45d85871f3ab72_type":"article","f48cdd8c7fc25a130c45d85871f3ab72_title":"Development of a gas-chromatographic method for simultaneous determination of cannabinoids and terpenes in hemp (Zeki\u010d and Kri\u017eman 2020)","f48cdd8c7fc25a130c45d85871f3ab72_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp","f48cdd8c7fc25a130c45d85871f3ab72_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Development of a gas-chromatographic method for simultaneous determination of cannabinoids and terpenes in hemp\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDevelopment of a gas-chromatographic method for simultaneous determination of cannabinoids and terpenes in hempJournal\n \nMoleculesAuthor(s)\n \nZeki\u010d, Jure; Kri\u017eman, MitjaAuthor affiliation(s)\n \nNational Institute of Chemistry - Ljubljana, University of LjubljanaPrimary contact\n \nEmail: mitja dot krizman at ki dot siYear published\n \n2020Volume and issue\n \n25(24)Article #\n \n5872DOI\n \n10.3390\/molecules25245872ISSN\n \n1420-3049Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/1420-3049\/25\/24\/5872\/htmDownload\n \nhttps:\/\/www.mdpi.com\/1420-3049\/25\/24\/5872\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Results and discussion \n\n3.1 Sample preparation \n3.2 Gas chromatographic separation \n3.3 Method performance and validation \n3.4 Comparison with HPLC \n\n\n4 Materials and methods \n\n4.1 Chemicals, reagents, and samples \n4.2 GC analysis \n4.3 HPLC analysis \n4.4 Standard solutions preparation \n4.5 Samples and preparation of extracts \n4.6 Quantitation, method precision, accuracy, sensitivity, linearity, and stability \n\n\n5 Conclusions \n6 Acknowledgements \n\n6.1 Author contributions \n6.2 Sample availability \n6.3 Funding \n6.4 Conflict of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nAn original gas-chromatographic method has been developed for simultaneous determination of major terpenes and cannabinoids in plant samples and their extracts. The main issues to be addressed were not only the large differences in polarity and volatility between both groups of analytes, but also the need for an exhaustive decarboxylation of cannabinoid acidic forms. Sample preparation was minimized by avoiding any analyte derivatization. Acetone was found to be the most appropriate extraction solvent. Successful chromatographic separation was achieved by using a medium-polarity column. Limits of detection ranged from 120 to 260 ng\/mL for terpenes and from 660 to 860 ng\/mL for cannabinoids. Parallel testing proved the results for cannabinoids are comparable to those obtained from established high-performance liquid chromatography (HPLC) methods. Despite very large differences in concentrations between both analyte groups, a linear range between 1 and 100 \u00b5g\/mL for terpenes and between 10 and 1500 \u00b5g\/mL for cannabinoids was determined.\nKeywords: cannabinoids, terpenes, cannabis, hemp, gas chromatography, capillary column\n\nIntroduction \nThe hemp plant (Cannabis sativa] and Cannabis indica), or simply Cannabis, is a plant that has elicited much interest throughout history because of its characteristics and various possibilities of use. Over the last few years, the popularity of the Cannabis plant and its constituents has particularly increased, and a widespread recognition of its usefulness, including for medical purposes, is becoming increasingly noticeable.[1][2][3][4][5] Hemp is known to contain various groups of compounds, probably the most characteristic among them being cannabinoids. Furthermore, cannabis also contains a diverse array of terpenes and flavonoids, as well as other groups of compounds.[6][7][8][9]\nCannabinoids are probably the most studied metabolites of cannabis. Many of their beneficial effects on human health are already known, and there is also a lot of ongoing research, discovering new ones.[10] As a result, the use of cannabinoids in a wide variety of preparations is growing, which is also reflected in increased cannabis production. At the same time, a need for an efficient, routine analytical method for monitoring the cannabinoid content in plant material has arisen. A number of methods for the analysis of cannabinoids in cannabis have indeed already been developed; among various approaches, the predominant is chromatographic analysis, in particular using gas chromatography (GC)[11][12][13][14][15][16][17][18] or high-performance liquid chromatography (HPLC).[16][19][20][21][22][23][24][25][26][27]\nEven though gas chromatography used to be the most common technique for analysis of cannabinoids in cannabis extracts, HPLC is currently increasingly gaining popularity in this field of application. HPLC determination of cannabinoids, in comparison to the analysis with GC, has some significant advantages: above all, it avoids the potential aggravating circumstances caused by the high temperature of analysis associated with GC, which affects the results mainly during the phase of sample injection and also indirectly during the analysis itself. Cannabinoids are found mainly in acidic forms in the plant, which eventually decarboxylate if they are exposed to raised temperature.[28] The temperature in the gas chromatograph also causes the process of decarboxylation, which is reflected in the results in two ways: we cannot separately determine acidic and decarboxylated forms of a particular cannabinoid, but only their total content. On the other hand, there is a significant probability that decarboxylation in the injector will not proceed completely.[29] Especially at higher cannabinoid concentrations, this may be reflected in apparently lower values measured and consequently irregular results of analysis. Both problems can be successfully solved by the derivatization of cannabinoids (including their acid forms) in the sample.[17][30][31][32] However, this represents an additional step that is often not desirable, because it increases probability for experimental error and prolongs analysis time, which may be a considerable drawback in terms of method suitability for routine use. With HPLC, all of these problems have been successfully avoided, as some relatively rapid, simple, and effective methods for the determination of both acidic and decarboxylated cannabinoids in cannabis samples have already been developed.[16][19][20][21][22][23][24][25][26][27][33]\nThus, two major approaches to chromatographic analysis of cannabinoids in hemp most often appear in the literature; direct analysis of a suitably diluted sample extract by liquid chromatography[16][19][20][21][22][23][24][25][26][27][33], or preliminary derivatization of the extract and subsequent analysis by gas chromatography.[17][30][31][32] Despite its mentioned drawbacks, the latter approach is still quite in use, somewhat for traditional reasons, but also for entirely practical reasons, since GC instrumentation is simpler and less expensive than HPLC, more economical for use, or maybe even the only option available.\nFor direct gas chromatographic analysis of cannabinoids, traditionally, the most commonly used stationary phase is 5% phenyl 95% dimethylpolysiloxane[7][16][18], followed by 100% dimethylpolysiloxane phase.[11][13] Recently, more polar stationary phases like 35% phenyl 65% dimethylpolysiloxane have also been used, with a potential gain in selectivity.[34][35]\nAnother relatively important group of compounds in cannabis are terpenes.[6][7][8][9] Different varieties of cannabis contain mainly different monoterpenes and sesquiterpenes, which also give a distinctive scent to hemp plants. From an analytical point of view, they are interesting because their profile is often characteristic of a particular variety or population of cannabis, which may enable identification of different plant specimens.[36][37] Terpenes in cannabis are also often credited for the so-called \u201centourage\u201d effect.[37] The analysis of terpenes is most often performed using gas chromatography as a separate type of analysis; successful separation and determination on different types of columns is usually quite fast, effective, and simple.\nAn unavoidable step in chromatographic analysis of plant material is analyte extraction from the sample. Both groups of compounds, terpenes and cannabinoids, can be extracted from the plant material by different approaches. For cannabinoids, the most common is classical extraction with a relatively apolar solvent (usually ethanol) either by mechanical shaking or by ultrasonic extraction. On the other hand, GC analysis of terpenes can also be done by the headspace sampling technique.[38] The alternative to headspace sampling is of course solvent extraction, in such a case a solvent of appropriate polarity must be chosen in relation to the analytes of interest. Terpenes and cannabinoids differ both in terms of volatility and polarity, as well as in the concentrations found in hemp samples. Terpene levels are usually significantly lower compared to cannabinoids. As demonstrated by Namdar et al.[7], an optimum solvent for terpene extraction was found to be a mixture of ethanol:hexane (3:7, v\/v), while for cannabinoids they corroborated the use of ethanol as the optimum solvent.\nGiven the above, a simultaneous analysis of terpenes and cannabinoids in hemp samples is not a trivial task. A notable example of such a type of methodology has been published by Franchina et al.[39] In that case, the methodology involved the use of sorptive extraction and thermal desorption sampling, two-dimensional gas chromatography, and mass detection. Such a methodology is certainly very detailed and useful when advanced studies have to be done, like in chemotaxonomy. For routine analyses, however, such a setup is probably too complicated and expensive. \nThe aim of this work was to find appropriate conditions, mainly in terms of sample preparation, for a simultaneous analysis of both groups of compounds, while keeping the overall experimental and instrumental setups simple.\n\nResults and discussion \nSample preparation \nAs already stated in the introduction, the main challenge in combined analysis of terpenes and cannabinoids is the sample preparation step, and more precisely the extraction conditions. As Namdar et al. have noted[7], optimum solvent composition for terpenes and cannabinoids differs. During their work, they found the mixture of ethanol:hexane (3:7, v\/v) to be the best compromise for extracting both groups of compounds. In this present work, the quest for a single solvent similar in properties to the mentioned work was undertaken. The solvent selection was then narrowed to acetone and ethyl acetate, with the added benefit of them being solvents with low environmental impact.[40] Finally, acetone was selected as the most appropriate solvent, based on the extraction recoveries obtained.\nAnother important parameter highly affecting the results is the sample-to-volume ratio during extraction. Besides having a direct effect on extraction efficiency as well, this ratio also has implications on the final analyte concentrations. Good overall analyte recoveries were obtained with sample-to-volume ratios between 1:10 and 1:25. During method development, it was found that a ratio of about 1:17 (i.e., 300 mg per 5 mL of solvent) was a good compromise between extraction efficiency while at the same time still providing sufficient concentrations of terpenes in working sample solutions in order to be quantified without concentrating the solution. Terpenes are unfortunately very volatile, and significant analyte losses can be expected with any of the solvent evaporation techniques.[7] Therefore, this sample preparation step was deliberately avoided. At the same time, cannabinoid concentrations in sample solutions proved to be below the upper practical limit in terms of detector linearity. Compared to terpenes, cannabinoids are more problematic to analyze. Besides their lower volatility, the possibility of incomplete decarboxylation of cannabinoid acidic forms during sample vaporization and injection must be prevented. This phenomenon is more pronounced at higher concentrations.[29]\nIn practice, the concentrations of major cannabinoids in the extracts were kept at up to 1.5 mg\/mL or below. By using such high cannabinoid concentrations, it was less challenging to quantify minor cannabinoids as well.\n\nGas chromatographic separation \nTerpenes and cannabinoids differ widely, both in terms of polarity and volatility. These two groups of compounds are therefore easily separated between each other using GC, although a wide temperature gradient program is needed due to a large difference in volatility. Successful separation of individual terpenes is not particularly challenging, as many works demonstrate.[7][8][36][38][41][42][43] The main challenge was to provide good separation of some cannabinoids, the most critical being the resolution between cannabichromene (CBC) and cannabidiol (CBD). Using the ubiquitous stationary phase based on 5% phenyl 95% dimethylpolysiloxane, the resolution between those two peaks proved to be unsuitable, since these two peaks overlapped, as proven by preliminary tests (data not published). Much better results are obtained using more polar stationary phases like 35% phenyl 65% dimethylpolysiloxane, as recent applications also demonstrate.[34][35] As a consequence, the choice for an even more polar stationary phase was made, namely 50% phenyl 50% dimethylpolysiloxane. According to initial expectations, a good resolution was obtained, and no issues related to overlapping of cannabinoid peaks were observed anymore. At the same time, using a relatively polar stationary phase did not impair the separation of terpenes. In fact, even more polar stationary phases were employed for terpenes.[41][42][43] Chromatograms of standard solutions and sample extracts are depicted on Figure 1, Figure 2. and Figure 3.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 Chromatograms (displayed tR = 13.5\u201317.0 min) of cannabinoid standard solution (top) and hemp plant (cannabigerol (CBG) chemotype) extract (bottom). Peak labelling: IS\u2014internal standard, CBC\u2014cannabichromene, CBD\u2014cannabidiol, \u03948-THC\u2014\u03948-tetrahydrocannabinol, \u03949-THC\u2014\u03949-tetrahydrocannabinol, CBG\u2014cannabigerol, CBN\u2014cannabinol.\n\n\n\n\n\n\n\n\n\n\n\n\n Fig. 2 Chromatograms (displayed tR = 3.0\u20139.5 min) of terpene standard solution (top) and hemp plant extract (bottom). *\u03b2-caryophyllene was identified by mass-spectrometric data.\n\n\n\n\n\n\n\n\n\n\n\n\n Fig. 3 Chromatogram of hemp sample extract (full scale).\n\n\n\nMethod performance and validation \nThe developed method exhibited good overall analytical performance within a relatively short analysis time, since it provided separation of two major groups of compounds in the cannabis plant. The most difficult to separate, CBD and CBC, were fully baseline resolved. Other analytes were also separated with excellent resolution. Validation and stability data (Table 1) confirmed the suitability of the method also for quantitative use. Sufficiently low sensitivity limits, which are important (especially from the terpenes standpoint), while also maintaining accurate results for higher cannabinoid concentrations, allow for the use of non-concentrated or non-diluted working sample solutions. This is a great advantage, especially in view of the process' simplicity.\n\n\n\n\n\n\n\nTable 1. Method validation parameters. LOD = limit of detection, LOQ = limit of quantification. a Determination of LOD and LOQ was based on extrapolation of signal-to-noise responses. Data in parentheses are validation parameters obtained with our HPLC method[33] for comparison purposes (where applicable). b Second consecutive extraction of samples gave no detectable peaks for the analyte.\n\n\nAnalyte\n\nInjection Precision (% RSD, n = 5)\n\nAccuracy (%)\n\nExtraction Efficiency (%)\n\nRepeatability (% RSD, n = 3)\n\nIntermediate Precision\n(% RSD, n = 9)\n\n\nLOD a (\u00b5g\/mL)\n\nLOQ a (\u00b5g\/mL)\n\nRegression Coefficient (R)\n\nStability 48 h (%)\n\n\ncannabidiol\n\n0.17\r\n(1.27)\n\n100.3\r\n(97.3)\n\n92.0\n\n0.08\r\n(0.78)\n\n1.59\r\n(1.27)\n\n0.662\r\n(0.093)\n\n2.207\r\n(0.310)\n\n0.9999\r\n(1.0000)\n\n99.9\r\n(102.7)\n\n\ncannabigerol\n\n1.60\r\n(0.52)\n\n97.6\r\n(96.9)\n\n92.1\n\n0.96\r\n(2.00)\n\n1.77\r\n(2.48)\n\n0.697\r\n(0.94)\n\n2.322\r\n(0.313)\n\n0.9999\r\n((0.9999)\n\n98.2\r\n(101.2)\n\n\n\u03948-tetrahydrocannabinol\n\n0.62\r\n(1.51)\n\n99.1\r\n(93.7)\n\n- b\n\n0.89\r\n(0.94)\n\n2.21\r\n(2.53)\n\n0.817\r\n(0.205)\n\n2.724\r\n(0.684)\n\n0.9999\r\n((0.9999)\n\n99.4\r\n(95.6)\n\n\n\u03949-tetrahydrocannabinol\n\n0.54\r\n(0.65)\n\n98.8\r\n(93.4)\n\n93.1\n\n0.42\r\n(0.85)\n\n2.35\r\n(2.34)\n\n0.822\r\n(0.196)\n\n2.739\r\n(0.654)\n\n1.0000\r\n(0.9999)\n\n99.6\r\n(94.0)\n\n\ncannabichromene\n\n0.79\r\n(0.18)\n\n100.1\r\n(113.3)\n\n92.0\n\n1.34\r\n(6.53)\n\n1.67\r\n(9.02)\n\n0.815\r\n(0.024)\n\n2.716\r\n(0.082)\n\n1.0000\r\n(0.9986)\n\n98.0\r\n(103.9)\n\n\ncannabinol\n\n1.27\r\n(0.20)\n\n98.9\r\n(88.4)\n\n- b\n\n1.25\r\n(3.38)\n\n1.88\r\n(3.86)\n\n0.857\r\n(0.007)\n\n2.858\r\n(0.023)\n\n0.9995\r\n(0.9999)\n\n100.7\r\n(103.8)\n\n\n\n\n\n\u03b1-pinene\n\n1.34\n\n99.6\n\n95.5\n\n0.54\n\n1.52\n\n0.259\n\n0.862\n\n0.9999\n\n99.8\n\n\n\u03b2-pinene\n\n0.65\n\n100.1\n\n95.2\n\n1.39\n\n1.33\n\n0.175\n\n0.585\n\n0.9999\n\n97.7\n\n\nmyrcene\n\n1.79\n\n100.2\n\n- b\n\n0.65\n\n3.15\n\n0.183\n\n0.609\n\n0.9998\n\n100.7\n\n\nlimonene\n\n1.05\n\n99.9\n\n95.9\n\n0.64\n\n2.89\n\n0.124\n\n0.412\n\n0.9999\n\n100.3\n\n\n\u03b1-terpinene\n\n0.62\n\n100.2\n\n- b\n\n1.54\n\n2.83\n\n0.193\n\n0.642\n\n1.0000\n\n97.0\n\n\n\u03b1-humulene\n\n0.75\n\n100.8\n\n93.2\n\n0.75\n\n1.60\n\n0.185\n\n0.616\n\n0.9999\n\n99.6\n\n\n\nComparison with HPLC \nIn order to confirm correctness of the results obtained from the newly developed GC method, HPLC analysis was performed, applying a previously published and validated method.[33] As mentioned before, GC analysis may not be sufficiently accurate mainly at higher cannabinoid concentrations. Therefore, HPLC quantification was performed for the two major cannabinoids in the samples\u2014cannabidiol or cannabigerol\u2014and their acidic forms which occur in rather high concentrations. The total content of these cannabinoids, measured by HPLC analysis, was in very high accordance with results obtained from GC analysis (Table 2). This also directly confirms the correctness of the developed GC method in this respect. Aside from being inherently a less sensitive methodology than HPLC[33], as depicted in Table 1, GC cannabinoid analysis is performance-wise fully comparable to HPLC. For most analytes, better repeatability figures were also obtained.\n\n\n\n\n\n\n\nTable 2. HPLC and GC method comparison. a Total cannabidiol\/cannabigerol are calculated as equimolar equivalents expressed in decarboxylated forms.\n\n\nHemp Sample Cultivar\n\nFedora 17\n\nCarmagnola\n\nFutura 75\n\nSanthica\n\n\nSample solution\n\n1\n\n2\n\n3\n\n4\n\n\nCannabidiolic acid (\u00b5g\/mL)\u2014HPLC\n\n298\n\n1110\n\n-\n\n-\n\n\nCannabidiol (\u00b5g\/mL)\u2014HPLC\n\n485\n\n468\n\n-\n\n-\n\n\nCannabigerolic acid (\u00b5g\/mL)\u2014HPLC\n\n-\n\n-\n\n428\n\n510\n\n\nCannabigerol (\u00b5g\/mL)\u2014HPLC\n\n-\n\n-\n\n118\n\n106\n\n\n\n\n\nHPLC\u2014total cannabidiola\n\n783\n\n1578\n\n-\n\n-\n\n\nHPLC\u2014total cannabigerola\n\n-\n\n-\n\n546\n\n616\n\n\nGC\u2014total cannabidiol\n\n839\n\n1635\n\n-\n\n-\n\n\nGC\u2014total cannabigerol\n\n-\n\n-\n\n605\n\n572\n\n\n\n\n\nCannabidiol\u2014relative difference (%)\n\n+7.1\n\n+3.6\n\n-\n\n-\n\n\nCannabigerol\u2014relative difference (%)\n\n-\n\n-\n\n+10.8\n\n\u22127.1\n\n\n\nMaterials and methods \nChemicals, reagents, and samples \nPractical grade ethyl acetate and n-hexane from Merck (Darmstadt, Germany) and acetone from Honeywell (Seelze, Germany) were used for standards preparation and sample treatment. HPLC grade methanol from Honeywell (Seelze, Germany) and deionised water from a Milli-Q apparatus (Millipore, Milford, MA, USA) were used to prepare mobile phase for HPLC analysis. Ammonium formate was LC\u2013MS grade (Sigma-Aldrich, St. Louis, MO, USA).\nTerpene reference standards of \u03b1-humulene (96%), limonene (97%), myrcene (>90%), \u03b1-pinene (99%), \u03b2-pinene (97%), and \u03b1-terpinene (95%) were obtained from Sigma-Aldrich (St. Louis, MO, USA).\nCannabinoid reference standards of cannabigerol (CBG, 99%) and cannabidiol (CBD, 99.9%) in solid form were obtained from LGC standards (Teddington, Middlesex, UK). \u03948-tetrahydrocannabinol (\u03948-THC), \u03949-tetrahydrocannabinol (\u03949-THC), cannabichromene (CBC), and cannabinol (CBN) were obtained as solution in methanol (1 mg\/mL) from LGC standards as well.\nSqualane (96%) was purchased from Sigma-Aldrich (St. Louis, MO, USA) and was applied as internal standard (IS).\nBuds of industrial hemp plant (stemless) material of various cultivars were obtained from local hemp growers.\n\nGC analysis \nAnalysis was performed by a Focus GC with a flame ionization detector (FID) (Thermo Scientific, Rodano, Milan, Italy) with a Rtx-50 capillary column (30 m \u00d7 0.25 mm \u00d7 0.25 \u00b5m film thickness) (Restek Corporation, Bellefonte, U.S.A.). One microlitre of sample was injected with a split ratio 30:1 at 310 \u00b0C using ultrapure grade helium as carrier gas at 2 mL\/min. The GC oven temperature program started at 60 \u00b0C (3 min), followed by a linear gradient of 20 \u00b0C\/min to 290 \u00b0C, temperature was then kept constant for 8 min. Flame ionisation detector temperature was 310 \u00b0C.\nThermo Electron Trace 2000 GC coupled with Thermo Electron DSQ quadrupole mass spectrometer (Thermo Scientific, Rodano, Milan, Italy) with an electron ionisation source at 70 eV in positive mode was applied for the purpose of terpene identification. Some parameters had to be adapted for gas chromatography\u2013mass spectrometry (GC-MS) analysis. Namely, 0.5 \u00b5L of sample was injected with a split ratio of 60:1, flow rate of carrier gas was 1 mL\/min. Other parameters remained the same as for GC-FID analysis. The MS data were acquired in full scan mode from m\/z 50\u2013450 with acquisition frequency of 4.2 scans per second.\n\nHPLC analysis \nAnalysis was performed according to Kri\u017eman's published method.[33] For HPLC analysis, sample solutions (as described below in \"Samples and preparation of extracts\") were further diluted with methanol 100-fold.\n\nStandard solutions preparation \nStock solutions of solid cannabinoid standards (cannabidiol and cannabigerol) were prepared in acetone in the concentration of 2.0 mg\/mL. Working standard solutions of cannabidiol and cannabigerol were prepared in the concentration range of 0.05 to 1.5 mg\/mL, and working standard solutions of cannabichromene, cannabinol, \u03948-tetrahydrocannabinol, and \u03949-tetrahydrocannabinol were prepared in the concentration range of 0.01 to 0.2 mg\/mL. All working solutions contained internal standard of concentration 0.13 mg\/mL and were prepared by diluting stock solution with acetone.\nStock solutions of terpene standards were prepared in the concentration of 2.0 mg\/mL in acetone, except for myrcene stock solution, which was prepared in hexane. Working standard solutions consisted of a mix of all terpene standards in the concentration range of 1.0 to 100 \u00b5g\/mL.\n\nSamples and preparation of extracts \nDried and powdered plant materials (300 mg) were extracted by sonication for 30 min at room temperature with acetone or ethyl acetate containing IS (130 \u00b5g\/mL). Sample solutions were then centrifuged at 16.000g for 10 min and the supernatant was transferred into GC vials.\n\nQuantitation, method precision, accuracy, sensitivity, linearity, and stability \nInjection precision was determined by five injections of working standard solution. Extraction efficiency was assessed by three consecutive extractions of selected plant samples, and then comparing the analyte recovery with the combined recovery of all extraction steps. Accuracy was determined by spiking sample solution with cannabinoid standards at concentration of 0.25 mg\/mL and terpene standards at concentration of 0.025 mg\/mL. Repeatability and intermediate precision were also tested on a homogeneous plant sample. Three replicates were assayed for repeatability, while three replicates were assayed on each of the three consecutive days for intermediate precision.\nLinearity was checked in the range of standard solutions concentration (0.05 to 1.5 mg\/mL for CBD and CBG; 0.01 to 0.2 mg\/mL for CBC, CBN, \u03948-THC, and \u03949-THC; and 0.001 to 0.1 mg\/mL for terpenes). Correlation coefficients were calculated with intercept values set at zero. Analyte peak areas were normalised by dividing them with IS peak areas. For stability tests, a sample solution was refrigerated at 4 \u00b0C in the dark for 48 h.\nIn order to confirm the correctness of the results, HPLC analysis of properly diluted sample extracts solutions in methanol was performed.\n\nConclusions \nA gas chromatography\u2013flame ionization detection (GC-FID) method for simultaneous analysis of terpenes and cannabinoids in hemp samples has been developed. The main issues concerning the method were ensuring appropriate sample preparation conditions for both terpenes and cannabinoids, successful separation of critical cannabinoid peaks, and quantitative decarboxylation of cannabinoid acidic forms. Acetone proved to be an appropriate solvent for quantitative extraction of all the analytes concerned, which occur in a wide concentration range, using a sufficiently low sample-to-solvent ratio. Separation-wise, the resolution between CBD and CBC peaks was substantially improved by using a relatively polar column with 50% phenyl 50% dimethylpolysiloxane stationary phase. Quantitative decarboxylation was ensured using a high injector temperature and low injection volumes. Peak identity was confirmed by GC-MS. Despite many of the method parameters being near the practical limits in terms of instrumentation capacity like temperature, detector response, etc., the method provides a robust tool for simultaneous quantitative analysis of these two chemically different groups of analytes.\n\nAcknowledgements \nJure Zeki\u010d is a student of the doctoral program in chemical sciences at the Faculty of Chemistry and Chemical Technology, University of Ljubljana, Ljubljana, Slovenia.\n\nAuthor contributions \nConceptualization, M.K. and J.Z.; investigation, J.Z.; supervision, M.K.; data analysis, J.Z. and M.K.; writing\u2014original draft preparation, J.Z.; writing\u2014review and editing, J.Z. and M.K. All authors have read and agreed to the published version of the manuscript.\n\nSample availability \nSamples of the compounds are available from the authors.\n\nFunding \nThis research was supported by the Slovenian Research Agency (research core funding No. P1-0005).\n\nConflict of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 Abuhasira, R.; Shbiro, L.; Landschaft, Y. (2018). \"Medical use of cannabis and cannabinoids containing products - Regulations in Europe and North America\". European Journal of Internal Medicine 49: 2\u20136. doi:10.1016\/j.ejim.2018.01.001. PMID 29329891.   \n\n\u2191 Hutchison, K.E.; Bidwell, L.C.; Ellingson, J.M. et al. (2019). \"Cannabis and Health Research: Rapid Progress Requires Innovative Research Designs\". Value in Health 22 (11): 1289\u201394. doi:10.1016\/j.jval.2019.05.005. PMID 31708066.   \n\n\u2191 Beauchet, O. (2018). \"Medical cannabis use in older patients: Update on medical knowledge\". Maturitas 118: 56\u20139. doi:10.1016\/j.maturitas.2018.10.010. PMID 30415756.   \n\n\u2191 Ebbert, J.O.; Scharf, E.L.; Hurt, R.T. (2018). \"Medical Cannabis\". Mayo Clinic Proceedings 93: 1842\u201347. doi:10.1016\/j.mayocp.2018.09.005. PMID 30522595.   \n\n\u2191 Blasco-Benito, S.; Seijo-Vila, M.; Caro-Villalobos, M. et al. (2018). \"Appraising the \"entourage effect\": Antitumor action of a pure cannabinoid versus a botanical drug preparation in preclinical models of breast cancer\". Biochemical Pharmacology 157: 285\u201393. doi:10.1016\/j.bcp.2018.06.025. PMID 29940172.   \n\n\u2191 6.0 6.1 Booth, J.K.; Bohlmann, J. (2019). \"Terpenes in Cannabis sativa - From plant genome to humans\". Plant Science 284: 67\u201372. doi:10.1016\/j.plantsci.2019.03.022. PMID 31084880.   \n\n\u2191 7.0 7.1 7.2 7.3 7.4 7.5 7.6 Namdar, D.; Mazuz, M.; Ion, A. et al. (2018). \"Variation in the compositions of cannabinoid and terpenoids in Cannabis sativa derived from inflorescence position along the stem and extraction methods\". Industrial Crops and Products 113: 376\u201382. doi:10.1016\/j.indcrop.2018.01.060.   \n\n\u2191 8.0 8.1 8.2 Delgado-Povedano, M.M.; Callado, C.S.-C.; Priego, Capote, F. et al. (2020). \"Untargeted characterization of extracts from Cannabis sativa L. cultivars by gas and liquid chromatography coupled to mass spectrometry in high resolution mode\". Talanta 208: 120384. doi:10.1016\/j.talanta.2019.120384.   \n\n\u2191 9.0 9.1 Nuutinen, T. (2018). \"Medicinal properties of terpenes found in Cannabis sativa and Humulus lupulus\". European Journal of Medicinal Chemistry 157: 198\u2013228. doi:10.1016\/j.ejmech.2018.07.076.   \n\n\u2191 Citti, C.; Linciano, P.; Russo, F. et al. (2019). \"A novel phytocannabinoid isolated from Cannabis sativa L. with an in vivo cannabimimetic activity higher than \u03949-tetrahydrocannabinol: \u03949-Tetrahydrocannabiphorol\". Scientific Reports 9: 20335. doi:10.1038\/s41598-019-56785-1.   \n\n\u2191 11.0 11.1 Gambaro, V.E.; Froldi, R.; Saligari, E. et al. (1995). \"A simple method for analyzing cannabis by gas chromatography\". Acta Toxicologica Argentina 3: 11\u201313.   \n\n\u2191 Gibson, C.R.; Williams, R.D.; Browder, R.O. (1998). \"Analysis of Hempen Ale for cannabinoids\". Journal of Analytical Toxicology 22 (2): 179. doi:10.1093\/jat\/22.2.179. PMID 9547418.   \n\n\u2191 13.0 13.1 Ross, S.A.; Mehmedic, Z.; Murphy, T.P. et al. (2008). \"GC-MS analysis of the total delta9-THC content of both drug- and fiber-type cannabis seeds\". Journal of Analytical Toxicology 24 (8): 715\u20137. doi:10.1093\/jat\/24.8.715. PMID 11110027.   \n\n\u2191 Raharjo, T.J.; Verpoorte, R. (2004). \"Methods for the analysis of cannabinoids in biological materials: A review\". Phytochemical Analysis 15 (2): 79\u201394. doi:10.1002\/pca.753. PMID 15116938.   \n\n\u2191 Pellegrini, M.; Marchei, E.; Pacifici, R. et al. (2005). \"A rapid and simple procedure for the determination of cannabinoids in hemp food products by gas chromatography-mass spectrometry\". Journal of Pharmaceutical and Biomedical Analysis 36 (5): 939\u201346. doi:10.1016\/j.jpba.2004.07.035.   \n\n\u2191 16.0 16.1 16.2 16.3 16.4 Gambaro, V.; Dell'Acqua, L.; Far\u00e8, F. et al. (2002). \"Determination of primary active constituents in Cannabis preparations by high-resolution gas chromatography\/flame ionization detection and high-performance liquid chromatography\/UV detection\". Analtica Chimica Acta 468 (2): 245-254. doi:10.1016\/S0003-2670(02)00660-8.   \n\n\u2191 17.0 17.1 17.2 Cardenia, V.; Toschi, T.G.; Scappini, S. et al. (2018). \"Development and validation of a Fast gas chromatography\/mass spectrometry method for the determination of cannabinoids in Cannabis sativa L\". Journal of Food and Drug Analysis 26 (4): 1283\u201392. doi:10.1016\/j.jfda.2018.06.001.   \n\n\u2191 18.0 18.1 United Nations Office on Drug and Crime (2009) (PDF). Recommended methods for the identification and analysis of cannabis and cannabis products. United Nations. ISBN 9789211482423. https:\/\/www.unodc.org\/documents\/scientific\/ST-NAR-40-Ebook_1.pdf . Retrieved 03 April 2020 .   \n\n\u2191 19.0 19.1 19.2 De Backer, B.; Debrus, B.; Lebrun, P. et al. (2009). \"Innovative development and validation of an HPLC\/DAD method for the qualitative and quantitative determination of major cannabinoids in cannabis plant material\". Journal of Chromatography B 877 (32): 4115-4124. doi:10.1016\/j.jchromb.2009.11.004.   \n\n\u2191 20.0 20.1 20.2 Aizpurua-Olaizola, O.; Omar, J.; Navarro, P. et al. (2014). \"Identification and quantification of cannabinoids in Cannabis sativa L. plants by high performance liquid chromatography-mass spectrometry\". Analytical and Bioanalytical Chemistry 406: 7549\u201360. doi:10.1007\/s00216-014-8177-x.   \n\n\u2191 21.0 21.1 21.2 Zgair, A.; Wong, J.C.M.; Sabri, A. et al. (2015). \"Development of a simple and sensitive HPLC\u2013UV method for the simultaneous determination of cannabidiol and \u03949-tetrahydrocannabinol in rat plasma\". Journal of Pharmaceutical and Biomedical Analysis 114: 145\u201351. doi:10.1016\/j.jpba.2015.05.019.   \n\n\u2191 22.0 22.1 22.2 Gul, W.; Gul, S.W.; Radwan, M.M. et al. (2015). \"Determination of 11 Cannabinoids in Biomass and Extracts of Different Varieties of Cannabis Using High-Performance Liquid Chromatography\". Journal of AOAC International 98 (6): 1523\u20138. doi:10.5740\/jaoacint.15-095. PMID 26651563.   \n\n\u2191 23.0 23.1 23.2 Giese, M.W.; Lewis, M.A.; Giese, L. et al. (2015). \"Development and Validation of a Reliable and Robust Method for the Analysis of Cannabinoids and Terpenes in Cannabis\". Journal of AOAC International 98 (6): 1503\u201322. doi:10.5740\/jaoacint.15-116. PMID 26651562.   \n\n\u2191 24.0 24.1 24.2 Citti, C.; Ciccarella, G.; Braghiroli, D. et al. (2016). \"Medicinal cannabis: Principal cannabinoids concentration and their stability evaluated by a high performance liquid chromatography coupled to diode array and quadrupole time of flight mass spectrometry method\". Journal of Pharmaceutical and Biomedical Analysis 128: 201\u20139. doi:10.1016\/j.jpba.2016.05.033. PMID 27268223.   \n\n\u2191 25.0 25.1 25.2 Patel, B.; Wene, D.; Fan, Z.T. (2017). \"Qualitative and quantitative measurement of cannabinoids in cannabis using modified HPLC\/DAD method\". Journal of Pharmaceutical and Biomedical Analysis 146: 15\u201323. doi:10.1016\/j.jpba.2017.07.021.   \n\n\u2191 26.0 26.1 26.2 Ciolino, L.A.; Ranieri, T.L.; Taylor, A.M. (2018). \"Commercial cannabis consumer products part 2: HPLC-DAD quantitative analysis of cannabis cannabinoids\". Forensic Science International 289: 438\u201347. doi:10.1016\/j.forsciint.2018.05.033.   \n\n\u2191 27.0 27.1 27.2 Mudge, E.M.; Murch, S.J.; Brown, P.N. (2017). \"Leaner and greener analysis of cannabinoids\". Analytical and Bioanalytical Chemistry 409: 3153\u201363. doi:10.1007\/s00216-017-0256-3.   \n\n\u2191 Wang, M.; Wang, Y.-H.; Avula, B. et al. (2016). \"Decarboxylation Study of Acidic Cannabinoids: A Novel Approach Using Ultra-High-Performance Supercritical Fluid Chromatography\/Photodiode Array-Mass Spectrometry\". Cannabis and Cannabinoid Research 1 (1): 262-271. doi:10.1089\/can.2016.0020.   \n\n\u2191 29.0 29.1 Dussy, F.E.; Hamberg, C.; Luginb\u00fchl, M. et al. (2005). \"Isolation of \u03949-THCA-A from hemp and analytical aspects concerning the determination of \u03949-THC in cannabis products\". Forensic Science International 149 (1): 3\u201310. doi:10.1016\/j.forsciint.2004.05.015.   \n\n\u2191 30.0 30.1 Kemp, P.M.; Abukhalaf, I.K.; Manno, J.E. et al. (2005). \"Cannabinoids in Humans. I. Analysis of \u03949-Tetrahydrocannabinol and Six Metabolites in Plasma and Urine Using GC-MS\". Journal of Analytical Toxicology 19 (5): 285\u201391. doi:10.1093\/jat\/19.5.285.   \n\n\u2191 31.0 31.1 Steinmeyer, S.; Bregel, D.; Warth, S. et al. (2002). \"Improved and validated method for the determination of \u03949-tetrahydrocannabinol (THC), 11-hydroxy-THC and 11-nor-9-carboxy-THC in serum, and in human liver microsomal preparations using gas chromatography\u2013mass spectrometry\". Journal of Chromatography B 772 (2): 239\u201348. doi:10.1016\/S1570-0232(02)00102-2.   \n\n\u2191 32.0 32.1 Leghissa, A.; Hildenbrand, Z.L.; Foss, F.W. et al. (2018). \"Determination of cannabinoids from a surrogate hops matrix using multiple reaction monitoring gas chromatography with triple quadrupole mass spectrometry\". Journal of Separation Science 41 (2): 459\u201368. doi:10.1002\/jssc.201700946.   \n\n\u2191 33.0 33.1 33.2 33.3 33.4 33.5 Kri\u017eman, M. (2019). \"A simplified approach for isocratic HPLC analysis of cannabinoids by fine tuning chromatographic selectivity\". European Food Research and Technology 246: 315\u201322. doi:10.1007\/s00217-019-03344-7.   \n\n\u2191 34.0 34.1 Phenomenex. \"Terpenes and Cannabinoids in Marijuana by GC\/FID on ZB-35HT\". Application Search by Name. https:\/\/www.phenomenex.com\/Application\/Detail\/22828 . Retrieved 18 March 2020 .   \n\n\u2191 35.0 35.1 Restek. \"Medical Cannabis Cannabinoids and Internal Standards on Rxi-35Sil MS\". Restek Searchable Chromatogram Library. https:\/\/www.restek.com\/chromatogram\/view\/GC_FS0549\/cannabinoids . Retrieved 18 March 2020 .   \n\n\u2191 36.0 36.1 Nissen, L.; Zatta, A.; Stefanini, I. et al. (2010). \"Characterization and antimicrobial activity of essential oils of industrial hemp varieties (Cannabis sativa L.)\". Fitoterapia 81 (5): 413\u20139. doi:10.1016\/j.fitote.2009.11.010. PMID 19969046.   \n\n\u2191 37.0 37.1 Russo, E.B. (2011). \"Taming THC: potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\". British Journal of Pharmacology 163 (7): 1344\u201364. doi:10.1111\/j.1476-5381.2011.01238.x. PMC PMC3165946. PMID 21749363. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946 .   \n\n\u2191 38.0 38.1 Osman, A.; Thorpe, J.W.; Caddy, B. (1985). \"Comparison of Cannabis Samples from Different Origins by the Headspace Technique and an Assessment of Chromatographic Traces Using the r-Matrix\". Journal of the Forensic Science Society 25 (6): 427\u201333. doi:10.1016\/S0015-7368(85)72431-0.   \n\n\u2191 Franchina, F.A.; Dubois, L.M.; Focant, J.-F. (2020). \"In-Depth Cannabis Multiclass Metabolite Profiling Using Sorptive Extraction and Multidimensional Gas Chromatography with Low- and High-Resolution Mass Spectrometry\". Analytical Chemistry 92 (15): 10512-10520. doi:10.1021\/acs.analchem.0c01301. PMID 32602704.   \n\n\u2191 Adams, D.J.; Dyson, P.J.; Tavener, S.J. (2004). Chemistry in Alternative Reaction Media. Wiley. p. 4. ISBN 0471498491.   \n\n\u2191 41.0 41.1 Restek. \"Terpenes in Blueberry Jack Medical Cannabis on Rxi-624Sil MS\". Restek Searchable Chromatogram Library. https:\/\/www.restek.com\/chromatogram\/view\/GC_FS0515 . Retrieved 07 April 2020 .   \n\n\u2191 42.0 42.1 Restek. \"Cannabis Terpenes Standard #1 & #2 on Rxi-624Sil MS\". Restek Searchable Chromatogram Library. https:\/\/www.restek.com\/chromatogram\/view\/GC_GN1200 . Retrieved 07 April 2020 .   \n\n\u2191 43.0 43.1 Fausett, A. (26 May 2020). \"Analysis of Terpene and Terpenoid Content in Cannabis Sativa Using Headspace with GC\/MSD\" (PDF). Agilent Technologies, Inc. https:\/\/www.agilent.com\/cs\/library\/applications\/application-terpenes-8890-gcms-headspace-5994-1497en-agilent.pdf . Retrieved 07 April 2020 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\">https:\/\/www.limswiki.org\/index.php\/Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on cannabis researchLIMSwiki journal articles on cannabis testing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 11 January 2021, at 23:20.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 232 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","f48cdd8c7fc25a130c45d85871f3ab72_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Development of a gas-chromatographic method for simultaneous determination of cannabinoids and terpenes in hemp<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>An original <a href=\"https:\/\/www.limswiki.org\/index.php\/Gas_chromatography\" title=\"Gas chromatography\" class=\"wiki-link\" data-key=\"e621fc6f90266fbc8db27d516e9cbb94\">gas-chromatographic<\/a> method has been developed for simultaneous determination of major <a href=\"https:\/\/en.wikipedia.org\/wiki\/Terpene\" class=\"extiw wiki-link\" title=\"wikipedia:Terpene\" data-key=\"5378b8663069e9faf390dc59f6cdd279\">terpenes<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabinoid\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabinoid\" data-key=\"fe5c4e73e0c21e16db393a214691296b\">cannabinoids<\/a> in plant <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">samples<\/a> and their <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_concentrate\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis concentrate\" data-key=\"e1754480a85316820c041e0f7f409adc\">extracts<\/a>. The main issues to be addressed were not only the large differences in polarity and volatility between both groups of analytes, but also the need for an exhaustive <a href=\"https:\/\/en.wikipedia.org\/wiki\/Decarboxylation\" class=\"extiw wiki-link\" title=\"wikipedia:Decarboxylation\" data-key=\"1b2f03c2feb1f0d2c530732788090504\">decarboxylation<\/a> of cannabinoid acidic forms. Sample preparation was minimized by avoiding any analyte derivatization. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Acetone\" class=\"extiw wiki-link\" title=\"wikipedia:Acetone\" data-key=\"5bbe34fa6bdf70e071834de67c6f292a\">Acetone<\/a> was found to be the most appropriate extraction <a href=\"https:\/\/en.wikipedia.org\/wiki\/Solvent\" class=\"extiw wiki-link\" title=\"wikipedia:Solvent\" data-key=\"7d1f3badf734642ed04f2202f534edf1\">solvent<\/a>. Successful chromatographic separation was achieved by using a medium-polarity column. Limits of detection ranged from 120 to 260 ng\/mL for terpenes and from 660 to 860 ng\/mL for cannabinoids. Parallel testing proved the results for cannabinoids are comparable to those obtained from established <a href=\"https:\/\/www.limswiki.org\/index.php\/High-performance_liquid_chromatography\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"dc274e995eb18113903eebaef69c0cf9\">high-performance liquid chromatography<\/a> (HPLC) methods. Despite very large differences in concentrations between both analyte groups, a linear range between 1 and 100 \u00b5g\/mL for terpenes and between 10 and 1500 \u00b5g\/mL for cannabinoids was determined.\n<\/p><p><b>Keywords<\/b>: cannabinoids, terpenes, cannabis, hemp, gas chromatography, capillary column\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>The <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hemp\" class=\"extiw wiki-link\" title=\"wikipedia:Hemp\" data-key=\"2e896be8a228178ae105f8b468061a0f\">hemp<\/a> plant (<i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_sativa\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis sativa\" data-key=\"618f17d4f6938557aacc017ee0f52bbd\">Cannabis sativa<\/a>]<\/i> and <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_indica\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis indica\" data-key=\"38fe6de66d0d5a463e61b05890cd336e\">Cannabis indica<\/a><\/i>), or simply <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis\" data-key=\"ae3a33525e4682427d4498e16c586f9e\">Cannabis<\/a><\/i>, is a plant that has elicited much interest throughout history because of its characteristics and various possibilities of use. Over the last few years, the popularity of the <i>Cannabis<\/i> plant and its constituents has particularly increased, and a widespread recognition of its usefulness, including for medical purposes, is becoming increasingly noticeable.<sup id=\"rdp-ebb-cite_ref-AbuharisaMedical18_1-0\" class=\"reference\"><a href=\"#cite_note-AbuharisaMedical18-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HutchisonCanna19_2-0\" class=\"reference\"><a href=\"#cite_note-HutchisonCanna19-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BeauchetMedical18_3-0\" class=\"reference\"><a href=\"#cite_note-BeauchetMedical18-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-EbbertMedical18_4-0\" class=\"reference\"><a href=\"#cite_note-EbbertMedical18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Blasco-BenitoAppr18_5-0\" class=\"reference\"><a href=\"#cite_note-Blasco-BenitoAppr18-5\">[5]<\/a><\/sup> Hemp is known to contain various groups of compounds, probably the most characteristic among them being <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabinoid\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabinoid\" data-key=\"fe5c4e73e0c21e16db393a214691296b\">cannabinoids<\/a>. Furthermore, cannabis also contains a diverse array of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Terpene\" class=\"extiw wiki-link\" title=\"wikipedia:Terpene\" data-key=\"5378b8663069e9faf390dc59f6cdd279\">terpenes<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Flavonoids\" class=\"extiw wiki-link\" title=\"wikipedia:Flavonoids\" data-key=\"42cfe142ae4ae5a1cfe4c818115456d4\">flavonoids<\/a>, as well as other groups of compounds.<sup id=\"rdp-ebb-cite_ref-BoothTerpenes19_6-0\" class=\"reference\"><a href=\"#cite_note-BoothTerpenes19-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NamdarVaria18_7-0\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Delgado-PovedanoUntarg20_8-0\" class=\"reference\"><a href=\"#cite_note-Delgado-PovedanoUntarg20-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NuutinenMedic18_9-0\" class=\"reference\"><a href=\"#cite_note-NuutinenMedic18-9\">[9]<\/a><\/sup>\n<\/p><p>Cannabinoids are probably the most studied metabolites of cannabis. Many of their beneficial effects on human health are already known, and there is also a lot of ongoing research, discovering new ones.<sup id=\"rdp-ebb-cite_ref-CittiANovel19_10-0\" class=\"reference\"><a href=\"#cite_note-CittiANovel19-10\">[10]<\/a><\/sup> As a result, the use of cannabinoids in a wide variety of preparations is growing, which is also reflected in increased cannabis production. At the same time, a need for an efficient, routine analytical method for monitoring the cannabinoid content in plant material has arisen. A number of methods for the analysis of cannabinoids in cannabis have indeed already been developed; among various approaches, the predominant is <a href=\"https:\/\/www.limswiki.org\/index.php\/Chromatography\" title=\"Chromatography\" class=\"wiki-link\" data-key=\"2615535d1f14c6cffdfad7285999ad9d\">chromatographic<\/a> analysis, in particular using <a href=\"https:\/\/www.limswiki.org\/index.php\/Gas_chromatography\" title=\"Gas chromatography\" class=\"wiki-link\" data-key=\"e621fc6f90266fbc8db27d516e9cbb94\">gas chromatography<\/a> (GC)<sup id=\"rdp-ebb-cite_ref-GambaroASimp95_11-0\" class=\"reference\"><a href=\"#cite_note-GambaroASimp95-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GibsonAnal98_12-0\" class=\"reference\"><a href=\"#cite_note-GibsonAnal98-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RossGC00_13-0\" class=\"reference\"><a href=\"#cite_note-RossGC00-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RaharjoMethods04_14-0\" class=\"reference\"><a href=\"#cite_note-RaharjoMethods04-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PellegriniARapid05_15-0\" class=\"reference\"><a href=\"#cite_note-PellegriniARapid05-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GambaroDeterm02_16-0\" class=\"reference\"><a href=\"#cite_note-GambaroDeterm02-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CardeniaDevelop18_17-0\" class=\"reference\"><a href=\"#cite_note-CardeniaDevelop18-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-UNODCRecomm09_18-0\" class=\"reference\"><a href=\"#cite_note-UNODCRecomm09-18\">[18]<\/a><\/sup> or <a href=\"https:\/\/www.limswiki.org\/index.php\/High-performance_liquid_chromatography\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"dc274e995eb18113903eebaef69c0cf9\">high-performance liquid chromatography<\/a> (HPLC).<sup id=\"rdp-ebb-cite_ref-GambaroDeterm02_16-1\" class=\"reference\"><a href=\"#cite_note-GambaroDeterm02-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeBackerInnov09_19-0\" class=\"reference\"><a href=\"#cite_note-DeBackerInnov09-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Aizpurua-OlaizolaIdent14_20-0\" class=\"reference\"><a href=\"#cite_note-Aizpurua-OlaizolaIdent14-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZgairDevelop15_21-0\" class=\"reference\"><a href=\"#cite_note-ZgairDevelop15-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GulDeterm15_22-0\" class=\"reference\"><a href=\"#cite_note-GulDeterm15-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GieseDevelop15_23-0\" class=\"reference\"><a href=\"#cite_note-GieseDevelop15-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CittiMedic16_24-0\" class=\"reference\"><a href=\"#cite_note-CittiMedic16-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PatelQual17_25-0\" class=\"reference\"><a href=\"#cite_note-PatelQual17-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CiolinoComm18_26-0\" class=\"reference\"><a href=\"#cite_note-CiolinoComm18-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MudgeLeaner17_27-0\" class=\"reference\"><a href=\"#cite_note-MudgeLeaner17-27\">[27]<\/a><\/sup>\n<\/p><p>Even though gas chromatography used to be the most common technique for analysis of cannabinoids in cannabis <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_concentrate\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis concentrate\" data-key=\"e1754480a85316820c041e0f7f409adc\">extracts<\/a>, HPLC is currently increasingly gaining popularity in this field of application. HPLC determination of cannabinoids, in comparison to the analysis with GC, has some significant advantages: above all, it avoids the potential aggravating circumstances caused by the high temperature of analysis associated with GC, which affects the results mainly during the phase of <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">sample<\/a> injection and also indirectly during the analysis itself. Cannabinoids are found mainly in acidic forms in the plant, which eventually <a href=\"https:\/\/en.wikipedia.org\/wiki\/Decarboxylation\" class=\"extiw wiki-link\" title=\"wikipedia:Decarboxylation\" data-key=\"1b2f03c2feb1f0d2c530732788090504\">decarboxylate<\/a> if they are exposed to raised temperature.<sup id=\"rdp-ebb-cite_ref-WangDecarb16_28-0\" class=\"reference\"><a href=\"#cite_note-WangDecarb16-28\">[28]<\/a><\/sup> The temperature in the gas chromatograph also causes the process of decarboxylation, which is reflected in the results in two ways: we cannot separately determine acidic and decarboxylated forms of a particular cannabinoid, but only their total content. On the other hand, there is a significant probability that decarboxylation in the injector will not proceed completely.<sup id=\"rdp-ebb-cite_ref-DussyIsol05_29-0\" class=\"reference\"><a href=\"#cite_note-DussyIsol05-29\">[29]<\/a><\/sup> Especially at higher cannabinoid concentrations, this may be reflected in apparently lower values measured and consequently irregular results of analysis. Both problems can be successfully solved by the derivatization of cannabinoids (including their acid forms) in the sample.<sup id=\"rdp-ebb-cite_ref-CardeniaDevelop18_17-1\" class=\"reference\"><a href=\"#cite_note-CardeniaDevelop18-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KempCanna95_30-0\" class=\"reference\"><a href=\"#cite_note-KempCanna95-30\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SteinmeyerImprov02_31-0\" class=\"reference\"><a href=\"#cite_note-SteinmeyerImprov02-31\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LeghissaDeterm18_32-0\" class=\"reference\"><a href=\"#cite_note-LeghissaDeterm18-32\">[32]<\/a><\/sup> However, this represents an additional step that is often not desirable, because it increases probability for experimental error and prolongs analysis time, which may be a considerable drawback in terms of method suitability for routine use. With HPLC, all of these problems have been successfully avoided, as some relatively rapid, simple, and effective methods for the determination of both acidic and decarboxylated cannabinoids in cannabis samples have already been developed.<sup id=\"rdp-ebb-cite_ref-GambaroDeterm02_16-2\" class=\"reference\"><a href=\"#cite_note-GambaroDeterm02-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeBackerInnov09_19-1\" class=\"reference\"><a href=\"#cite_note-DeBackerInnov09-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Aizpurua-OlaizolaIdent14_20-1\" class=\"reference\"><a href=\"#cite_note-Aizpurua-OlaizolaIdent14-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZgairDevelop15_21-1\" class=\"reference\"><a href=\"#cite_note-ZgairDevelop15-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GulDeterm15_22-1\" class=\"reference\"><a href=\"#cite_note-GulDeterm15-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GieseDevelop15_23-1\" class=\"reference\"><a href=\"#cite_note-GieseDevelop15-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CittiMedic16_24-1\" class=\"reference\"><a href=\"#cite_note-CittiMedic16-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PatelQual17_25-1\" class=\"reference\"><a href=\"#cite_note-PatelQual17-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CiolinoComm18_26-1\" class=\"reference\"><a href=\"#cite_note-CiolinoComm18-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MudgeLeaner17_27-1\" class=\"reference\"><a href=\"#cite_note-MudgeLeaner17-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Kri.C5.BEmanASimp19_33-0\" class=\"reference\"><a href=\"#cite_note-Kri.C5.BEmanASimp19-33\">[33]<\/a><\/sup>\n<\/p><p>Thus, two major approaches to chromatographic analysis of cannabinoids in hemp most often appear in the literature; direct analysis of a suitably diluted sample extract by liquid chromatography<sup id=\"rdp-ebb-cite_ref-GambaroDeterm02_16-3\" class=\"reference\"><a href=\"#cite_note-GambaroDeterm02-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeBackerInnov09_19-2\" class=\"reference\"><a href=\"#cite_note-DeBackerInnov09-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Aizpurua-OlaizolaIdent14_20-2\" class=\"reference\"><a href=\"#cite_note-Aizpurua-OlaizolaIdent14-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZgairDevelop15_21-2\" class=\"reference\"><a href=\"#cite_note-ZgairDevelop15-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GulDeterm15_22-2\" class=\"reference\"><a href=\"#cite_note-GulDeterm15-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GieseDevelop15_23-2\" class=\"reference\"><a href=\"#cite_note-GieseDevelop15-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CittiMedic16_24-2\" class=\"reference\"><a href=\"#cite_note-CittiMedic16-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PatelQual17_25-2\" class=\"reference\"><a href=\"#cite_note-PatelQual17-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CiolinoComm18_26-2\" class=\"reference\"><a href=\"#cite_note-CiolinoComm18-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MudgeLeaner17_27-2\" class=\"reference\"><a href=\"#cite_note-MudgeLeaner17-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Kri.C5.BEmanASimp19_33-1\" class=\"reference\"><a href=\"#cite_note-Kri.C5.BEmanASimp19-33\">[33]<\/a><\/sup>, or preliminary derivatization of the extract and subsequent analysis by gas chromatography.<sup id=\"rdp-ebb-cite_ref-CardeniaDevelop18_17-2\" class=\"reference\"><a href=\"#cite_note-CardeniaDevelop18-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KempCanna95_30-1\" class=\"reference\"><a href=\"#cite_note-KempCanna95-30\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SteinmeyerImprov02_31-1\" class=\"reference\"><a href=\"#cite_note-SteinmeyerImprov02-31\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LeghissaDeterm18_32-1\" class=\"reference\"><a href=\"#cite_note-LeghissaDeterm18-32\">[32]<\/a><\/sup> Despite its mentioned drawbacks, the latter approach is still quite in use, somewhat for traditional reasons, but also for entirely practical reasons, since GC instrumentation is simpler and less expensive than HPLC, more economical for use, or maybe even the only option available.\n<\/p><p>For direct gas chromatographic analysis of cannabinoids, traditionally, the most commonly used stationary phase is 5% phenyl 95% dimethylpolysiloxane<sup id=\"rdp-ebb-cite_ref-NamdarVaria18_7-1\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GambaroDeterm02_16-4\" class=\"reference\"><a href=\"#cite_note-GambaroDeterm02-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-UNODCRecomm09_18-1\" class=\"reference\"><a href=\"#cite_note-UNODCRecomm09-18\">[18]<\/a><\/sup>, followed by 100% dimethylpolysiloxane phase.<sup id=\"rdp-ebb-cite_ref-GambaroASimp95_11-1\" class=\"reference\"><a href=\"#cite_note-GambaroASimp95-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RossGC00_13-1\" class=\"reference\"><a href=\"#cite_note-RossGC00-13\">[13]<\/a><\/sup> Recently, more polar stationary phases like 35% phenyl 65% dimethylpolysiloxane have also been used, with a potential gain in selectivity.<sup id=\"rdp-ebb-cite_ref-PhenomenexZB-35HT_34-0\" class=\"reference\"><a href=\"#cite_note-PhenomenexZB-35HT-34\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RestekRxi-35Sil_35-0\" class=\"reference\"><a href=\"#cite_note-RestekRxi-35Sil-35\">[35]<\/a><\/sup>\n<\/p><p>Another relatively important group of compounds in cannabis are terpenes.<sup id=\"rdp-ebb-cite_ref-BoothTerpenes19_6-1\" class=\"reference\"><a href=\"#cite_note-BoothTerpenes19-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NamdarVaria18_7-2\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Delgado-PovedanoUntarg20_8-1\" class=\"reference\"><a href=\"#cite_note-Delgado-PovedanoUntarg20-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NuutinenMedic18_9-1\" class=\"reference\"><a href=\"#cite_note-NuutinenMedic18-9\">[9]<\/a><\/sup> Different varieties of cannabis contain mainly different monoterpenes and sesquiterpenes, which also give a distinctive scent to hemp plants. From an analytical point of view, they are interesting because their profile is often characteristic of a particular variety or population of cannabis, which may enable identification of different plant specimens.<sup id=\"rdp-ebb-cite_ref-NissenCharact10_36-0\" class=\"reference\"><a href=\"#cite_note-NissenCharact10-36\">[36]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RussoTaming11_37-0\" class=\"reference\"><a href=\"#cite_note-RussoTaming11-37\">[37]<\/a><\/sup> Terpenes in cannabis are also often credited for the so-called \u201centourage\u201d effect.<sup id=\"rdp-ebb-cite_ref-RussoTaming11_37-1\" class=\"reference\"><a href=\"#cite_note-RussoTaming11-37\">[37]<\/a><\/sup> The analysis of terpenes is most often performed using gas chromatography as a separate type of analysis; successful separation and determination on different types of columns is usually quite fast, effective, and simple.\n<\/p><p>An unavoidable step in chromatographic analysis of plant material is analyte extraction from the sample. Both groups of compounds, terpenes and cannabinoids, can be extracted from the plant material by different approaches. For cannabinoids, the most common is classical extraction with a relatively apolar solvent (usually ethanol) either by mechanical shaking or by ultrasonic extraction. On the other hand, GC analysis of terpenes can also be done by the headspace sampling technique.<sup id=\"rdp-ebb-cite_ref-OsmanCompar85_38-0\" class=\"reference\"><a href=\"#cite_note-OsmanCompar85-38\">[38]<\/a><\/sup> The alternative to headspace sampling is of course solvent extraction, in such a case a solvent of appropriate polarity must be chosen in relation to the analytes of interest. Terpenes and cannabinoids differ both in terms of volatility and polarity, as well as in the concentrations found in hemp samples. Terpene levels are usually significantly lower compared to cannabinoids. As demonstrated by Namdar <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-NamdarVaria18_7-3\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-7\">[7]<\/a><\/sup>, an optimum solvent for terpene extraction was found to be a mixture of ethanol:hexane (3:7, <i>v\/v<\/i>), while for cannabinoids they corroborated the use of ethanol as the optimum solvent.\n<\/p><p>Given the above, a simultaneous analysis of terpenes and cannabinoids in hemp samples is not a trivial task. A notable example of such a type of methodology has been published by Franchina <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-FranchinaInDepth20_39-0\" class=\"reference\"><a href=\"#cite_note-FranchinaInDepth20-39\">[39]<\/a><\/sup> In that case, the methodology involved the use of sorptive extraction and thermal desorption sampling, two-dimensional gas chromatography, and mass detection. Such a methodology is certainly very detailed and useful when advanced studies have to be done, like in chemotaxonomy. For routine analyses, however, such a setup is probably too complicated and expensive. \n<\/p><p>The aim of this work was to find appropriate conditions, mainly in terms of sample preparation, for a simultaneous analysis of both groups of compounds, while keeping the overall experimental and instrumental setups simple.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Sample_preparation\">Sample preparation<\/span><\/h3>\n<p>As already stated in the introduction, the main challenge in combined analysis of terpenes and cannabinoids is the sample preparation step, and more precisely the extraction conditions. As Namdar <i>et al.<\/i> have noted<sup id=\"rdp-ebb-cite_ref-NamdarVaria18_7-4\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-7\">[7]<\/a><\/sup>, optimum solvent composition for terpenes and cannabinoids differs. During their work, they found the mixture of ethanol:hexane (3:7, <i>v\/v<\/i>) to be the best compromise for extracting both groups of compounds. In this present work, the quest for a single solvent similar in properties to the mentioned work was undertaken. The solvent selection was then narrowed to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Acetone\" class=\"extiw wiki-link\" title=\"wikipedia:Acetone\" data-key=\"5bbe34fa6bdf70e071834de67c6f292a\">acetone<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Ethyl_acetate\" class=\"extiw wiki-link\" title=\"wikipedia:Ethyl acetate\" data-key=\"14686ddc27bd5cf5fb6661666eed6001\">ethyl acetate<\/a>, with the added benefit of them being solvents with low environmental impact.<sup id=\"rdp-ebb-cite_ref-AdamsChemist04_40-0\" class=\"reference\"><a href=\"#cite_note-AdamsChemist04-40\">[40]<\/a><\/sup> Finally, acetone was selected as the most appropriate solvent, based on the extraction recoveries obtained.\n<\/p><p>Another important parameter highly affecting the results is the sample-to-volume ratio during extraction. Besides having a direct effect on extraction efficiency as well, this ratio also has implications on the final analyte concentrations. Good overall analyte recoveries were obtained with sample-to-volume ratios between 1:10 and 1:25. During method development, it was found that a ratio of about 1:17 (i.e., 300 mg per 5 mL of solvent) was a good compromise between extraction efficiency while at the same time still providing sufficient concentrations of terpenes in working sample solutions in order to be quantified without concentrating the solution. Terpenes are unfortunately very volatile, and significant analyte losses can be expected with any of the solvent evaporation techniques.<sup id=\"rdp-ebb-cite_ref-NamdarVaria18_7-5\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-7\">[7]<\/a><\/sup> Therefore, this sample preparation step was deliberately avoided. At the same time, cannabinoid concentrations in sample solutions proved to be below the upper practical limit in terms of detector linearity. Compared to terpenes, cannabinoids are more problematic to analyze. Besides their lower volatility, the possibility of incomplete decarboxylation of cannabinoid acidic forms during sample vaporization and injection must be prevented. This phenomenon is more pronounced at higher concentrations.<sup id=\"rdp-ebb-cite_ref-DussyIsol05_29-1\" class=\"reference\"><a href=\"#cite_note-DussyIsol05-29\">[29]<\/a><\/sup>\n<\/p><p>In practice, the concentrations of major cannabinoids in the extracts were kept at up to 1.5 mg\/mL or below. By using such high cannabinoid concentrations, it was less challenging to quantify minor cannabinoids as well.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Gas_chromatographic_separation\">Gas chromatographic separation<\/span><\/h3>\n<p>Terpenes and cannabinoids differ widely, both in terms of polarity and volatility. These two groups of compounds are therefore easily separated between each other using GC, although a wide temperature gradient program is needed due to a large difference in volatility. Successful separation of individual terpenes is not particularly challenging, as many works demonstrate.<sup id=\"rdp-ebb-cite_ref-NamdarVaria18_7-6\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Delgado-PovedanoUntarg20_8-2\" class=\"reference\"><a href=\"#cite_note-Delgado-PovedanoUntarg20-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NissenCharact10_36-1\" class=\"reference\"><a href=\"#cite_note-NissenCharact10-36\">[36]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OsmanCompar85_38-1\" class=\"reference\"><a href=\"#cite_note-OsmanCompar85-38\">[38]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RestekRxi-624Sil_41-0\" class=\"reference\"><a href=\"#cite_note-RestekRxi-624Sil-41\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RestekRxi-624SilStand1_2_42-0\" class=\"reference\"><a href=\"#cite_note-RestekRxi-624SilStand1_2-42\">[42]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FausettAnal20_43-0\" class=\"reference\"><a href=\"#cite_note-FausettAnal20-43\">[43]<\/a><\/sup> The main challenge was to provide good separation of some cannabinoids, the most critical being the resolution between <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabichromene\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabichromene\" data-key=\"2f3d934372efdfa61bdc2db408011a6c\">cannabichromene<\/a> (CBC) and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabidiol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabidiol\" data-key=\"0c46162c3d7b370d7448646c25334265\">cannabidiol<\/a> (CBD). Using the ubiquitous stationary phase based on 5% phenyl 95% dimethylpolysiloxane, the resolution between those two peaks proved to be unsuitable, since these two peaks overlapped, as proven by preliminary tests (data not published). Much better results are obtained using more polar stationary phases like 35% phenyl 65% dimethylpolysiloxane, as recent applications also demonstrate.<sup id=\"rdp-ebb-cite_ref-PhenomenexZB-35HT_34-1\" class=\"reference\"><a href=\"#cite_note-PhenomenexZB-35HT-34\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RestekRxi-35Sil_35-1\" class=\"reference\"><a href=\"#cite_note-RestekRxi-35Sil-35\">[35]<\/a><\/sup> As a consequence, the choice for an even more polar stationary phase was made, namely 50% phenyl 50% dimethylpolysiloxane. According to initial expectations, a good resolution was obtained, and no issues related to overlapping of cannabinoid peaks were observed anymore. At the same time, using a relatively polar stationary phase did not impair the separation of terpenes. In fact, even more polar stationary phases were employed for terpenes.<sup id=\"rdp-ebb-cite_ref-RestekRxi-624Sil_41-1\" class=\"reference\"><a href=\"#cite_note-RestekRxi-624Sil-41\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RestekRxi-624SilStand1_2_42-1\" class=\"reference\"><a href=\"#cite_note-RestekRxi-624SilStand1_2-42\">[42]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FausettAnal20_43-1\" class=\"reference\"><a href=\"#cite_note-FausettAnal20-43\">[43]<\/a><\/sup> Chromatograms of standard solutions and sample extracts are depicted on Figure 1, Figure 2. and Figure 3.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Zeki%C4%8D_Molecules2020_25-24.png\" class=\"image wiki-link\" data-key=\"f7852fee8878dc285a64ed433215bef8\"><img alt=\"Fig1 Zeki\u010d Molecules2020 25-24.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3b\/Fig1_Zeki%C4%8D_Molecules2020_25-24.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> Chromatograms (displayed tR = 13.5\u201317.0 min) of cannabinoid standard solution (<b>top<\/b>) and hemp plant (cannabigerol (CBG) chemotype) extract (<b>bottom<\/b>). Peak labelling: IS\u2014internal standard, CBC\u2014cannabichromene, CBD\u2014cannabidiol, \u03948-THC\u2014\u03948-tetrahydrocannabinol, \u03949-THC\u2014\u03949-tetrahydrocannabinol, CBG\u2014cannabigerol, CBN\u2014cannabinol.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Zeki%C4%8D_Molecules2020_25-24.png\" class=\"image wiki-link\" data-key=\"e2a2375c56e8d2519980b2a4fd21e946\"><img alt=\"Fig2 Zeki\u010d Molecules2020 25-24.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b9\/Fig2_Zeki%C4%8D_Molecules2020_25-24.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> Chromatograms (displayed tR = 3.0\u20139.5 min) of terpene standard solution (<b>top<\/b>) and hemp plant extract (<b>bottom<\/b>). *\u03b2-caryophyllene was identified by mass-spectrometric data.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Zeki%C4%8D_Molecules2020_25-24.png\" class=\"image wiki-link\" data-key=\"b1468d22884ceaca23dc3ead8f55dee0\"><img alt=\"Fig3 Zeki\u010d Molecules2020 25-24.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/da\/Fig3_Zeki%C4%8D_Molecules2020_25-24.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3<\/b> Chromatogram of hemp sample extract (full scale).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Method_performance_and_validation\">Method performance and validation<\/span><\/h3>\n<p>The developed method exhibited good overall analytical performance within a relatively short analysis time, since it provided separation of two major groups of compounds in the cannabis plant. The most difficult to separate, CBD and CBC, were fully baseline resolved. Other analytes were also separated with excellent resolution. Validation and stability data (Table 1) confirmed the suitability of the method also for quantitative use. Sufficiently low sensitivity limits, which are important (especially from the terpenes standpoint), while also maintaining accurate results for higher cannabinoid concentrations, allow for the use of non-concentrated or non-diluted working sample solutions. This is a great advantage, especially in view of the process' simplicity.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"10\"><b>Table 1.<\/b> Method validation parameters. LOD = limit of detection, LOQ = limit of quantification. <sup>a<\/sup> Determination of LOD and LOQ was based on extrapolation of signal-to-noise responses. Data in parentheses are validation parameters obtained with our HPLC method<sup id=\"rdp-ebb-cite_ref-Kri.C5.BEmanASimp19_33-2\" class=\"reference\"><a href=\"#cite_note-Kri.C5.BEmanASimp19-33\">[33]<\/a><\/sup> for comparison purposes (where applicable). <sup>b<\/sup> Second consecutive extraction of samples gave no detectable peaks for the analyte.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Analyte\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Injection Precision (% RSD, <i>n<\/i> = 5)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Accuracy (%)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Extraction Efficiency (%)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Repeatability (% RSD, <i>n<\/i> = 3)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Intermediate Precision\n<p>(% RSD, <i>n<\/i> = 9)\n<\/p>\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">LOD <sup>a<\/sup> (\u00b5g\/mL)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">LOQ <sup>a<\/sup> (\u00b5g\/mL)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Regression Coefficient (<i>R<\/i>)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Stability 48 h (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">cannabidiol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.17<br \/>(1.27)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.3<br \/>(97.3)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">92.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08<br \/>(0.78)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.59<br \/>(1.27)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.662<br \/>(0.093)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.207<br \/>(0.310)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9999<br \/>(1.0000)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.9<br \/>(102.7)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabigerol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabigerol\" data-key=\"ff3a6dcc74cd7fecd03bf2b259b03105\">cannabigerol<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.60<br \/>(0.52)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97.6<br \/>(96.9)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">92.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.96<br \/>(2.00)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.77<br \/>(2.48)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.697<br \/>(0.94)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.322<br \/>(0.313)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9999<br \/>((0.9999)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.2<br \/>(101.2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Tetrahydrocannabinol\" class=\"extiw wiki-link\" title=\"wikipedia:Tetrahydrocannabinol\" data-key=\"c63b7f849adf168f4b4ff293132f1e53\">\u0394<sup>8<\/sup>-tetrahydrocannabinol<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62<br \/>(1.51)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.1<br \/>(93.7)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">- <sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.89<br \/>(0.94)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.21<br \/>(2.53)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.817<br \/>(0.205)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.724<br \/>(0.684)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9999<br \/>((0.9999)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.4<br \/>(95.6)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u0394<sup>9<\/sup>-tetrahydrocannabinol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.54<br \/>(0.65)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.8<br \/>(93.4)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">93.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.42<br \/>(0.85)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.35<br \/>(2.34)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.822<br \/>(0.196)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.739<br \/>(0.654)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0000<br \/>(0.9999)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.6<br \/>(94.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">cannabichromene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.79<br \/>(0.18)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.1<br \/>(113.3)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">92.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.34<br \/>(6.53)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.67<br \/>(9.02)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.815<br \/>(0.024)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.716<br \/>(0.082)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0000<br \/>(0.9986)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.0<br \/>(103.9)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabinol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabinol\" data-key=\"8fa80479b70a31dacda0d3de9d8880ad\">cannabinol<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.27<br \/>(0.20)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.9<br \/>(88.4)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">- <sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.25<br \/>(3.38)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.88<br \/>(3.86)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.857<br \/>(0.007)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.858<br \/>(0.023)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9995<br \/>(0.9999)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.7<br \/>(103.8)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"10\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Pinene\" class=\"extiw wiki-link\" title=\"wikipedia:Pinene\" data-key=\"7533780ef87d647364d71522d0f6382b\">\u03b1-pinene<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">95.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.54\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.52\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.259\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.862\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9999\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b2-pinene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.65\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">95.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.175\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.585\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9999\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97.7\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Myrcene\" class=\"extiw wiki-link\" title=\"wikipedia:Myrcene\" data-key=\"7404d3e93d64a18382f7755500e23832\">myrcene<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.79\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">- <sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.65\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.183\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.609\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9998\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.7\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Limonene\" class=\"extiw wiki-link\" title=\"wikipedia:Limonene\" data-key=\"bf26891b1a237335f3557c816407a12f\">limonene<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">95.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.64\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.89\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.124\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.412\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9999\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Terpinene\" class=\"extiw wiki-link\" title=\"wikipedia:Terpinene\" data-key=\"a4a902770918909cc0c8f0db4c801c37\">\u03b1-terpinene<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">- <sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.54\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.83\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.193\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.642\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Humulene\" class=\"extiw wiki-link\" title=\"wikipedia:Humulene\" data-key=\"7b0eeded2634a2bc7d1989f318518b9b\">\u03b1-humulene<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.75\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">93.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.75\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.60\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.185\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.616\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9999\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">99.6\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Comparison_with_HPLC\">Comparison with HPLC<\/span><\/h3>\n<p>In order to confirm correctness of the results obtained from the newly developed GC method, HPLC analysis was performed, applying a previously published and validated method.<sup id=\"rdp-ebb-cite_ref-Kri.C5.BEmanASimp19_33-3\" class=\"reference\"><a href=\"#cite_note-Kri.C5.BEmanASimp19-33\">[33]<\/a><\/sup> As mentioned before, GC analysis may not be sufficiently accurate mainly at higher cannabinoid concentrations. Therefore, HPLC quantification was performed for the two major cannabinoids in the samples\u2014cannabidiol or cannabigerol\u2014and their acidic forms which occur in rather high concentrations. The total content of these cannabinoids, measured by HPLC analysis, was in very high accordance with results obtained from GC analysis (Table 2). This also directly confirms the correctness of the developed GC method in this respect. Aside from being inherently a less sensitive methodology than HPLC<sup id=\"rdp-ebb-cite_ref-Kri.C5.BEmanASimp19_33-4\" class=\"reference\"><a href=\"#cite_note-Kri.C5.BEmanASimp19-33\">[33]<\/a><\/sup>, as depicted in Table 1, GC cannabinoid analysis is performance-wise fully comparable to HPLC. For most analytes, better repeatability figures were also obtained.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 2.<\/b> HPLC and GC method comparison. <sup>a<\/sup> Total cannabidiol\/cannabigerol are calculated as equimolar equivalents expressed in decarboxylated forms.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Hemp Sample Cultivar\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Fedora 17\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Carmagnola\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Futura 75\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Santhica\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Sample solution\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">1\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">2\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">3\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">4\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabidiolic_acid\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabidiolic acid\" data-key=\"92e4e91322213e9a82ecd8ca7647526d\">Cannabidiolic acid<\/a> (\u00b5g\/mL)\u2014HPLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">298\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1110\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cannabidiol (\u00b5g\/mL)\u2014HPLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">485\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">468\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabigerolic_acid\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabigerolic acid\" data-key=\"959b7e5353aec2e1ce54cf9e4273b606\">Cannabigerolic acid<\/a> (\u00b5g\/mL)\u2014HPLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">428\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">510\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cannabigerol (\u00b5g\/mL)\u2014HPLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">118\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">106\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"10\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HPLC\u2014total cannabidiol<sup>a<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">783\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1578\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HPLC\u2014total cannabigerol<sup>a<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">546\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">616\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GC\u2014total cannabidiol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">839\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1635\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GC\u2014total cannabigerol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">605\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">572\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"10\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cannabidiol\u2014relative difference (%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+7.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+3.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cannabigerol\u2014relative difference (%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">+10.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u22127.1\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Chemicals.2C_reagents.2C_and_samples\">Chemicals, reagents, and samples<\/span><\/h3>\n<p>Practical grade ethyl acetate and n-hexane from Merck (Darmstadt, Germany) and acetone from Honeywell (Seelze, Germany) were used for standards preparation and sample treatment. HPLC grade methanol from Honeywell (Seelze, Germany) and deionised water from a Milli-Q apparatus (Millipore, Milford, MA, USA) were used to prepare mobile phase for HPLC analysis. Ammonium formate was LC\u2013MS grade (Sigma-Aldrich, St. Louis, MO, USA).\n<\/p><p>Terpene reference standards of \u03b1-humulene (96%), limonene (97%), myrcene (>90%), \u03b1-pinene (99%), \u03b2-pinene (97%), and \u03b1-terpinene (95%) were obtained from Sigma-Aldrich (St. Louis, MO, USA).\n<\/p><p>Cannabinoid reference standards of cannabigerol (CBG, 99%) and cannabidiol (CBD, 99.9%) in solid form were obtained from LGC standards (Teddington, Middlesex, UK). \u03948-tetrahydrocannabinol (\u03948-THC), \u03949-tetrahydrocannabinol (\u03949-THC), cannabichromene (CBC), and cannabinol (CBN) were obtained as solution in methanol (1 mg\/mL) from LGC standards as well.\n<\/p><p>Squalane (96%) was purchased from Sigma-Aldrich (St. Louis, MO, USA) and was applied as internal standard (IS).\n<\/p><p>Buds of industrial hemp plant (stemless) material of various cultivars were obtained from local hemp growers.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"GC_analysis\">GC analysis<\/span><\/h3>\n<p>Analysis was performed by a Focus GC with a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Flame_ionization_detector\" class=\"extiw wiki-link\" title=\"wikipedia:Flame ionization detector\" data-key=\"1fa7a3fb0a27b7eb5ce68fc5644dc108\">flame ionization detector<\/a> (FID) (Thermo Scientific, Rodano, Milan, Italy) with a Rtx-50 capillary column (30 m \u00d7 0.25 mm \u00d7 0.25 \u00b5m film thickness) (Restek Corporation, Bellefonte, U.S.A.). One microlitre of sample was injected with a split ratio 30:1 at 310 \u00b0C using ultrapure grade helium as carrier gas at 2 mL\/min. The GC oven temperature program started at 60 \u00b0C (3 min), followed by a linear gradient of 20 \u00b0C\/min to 290 \u00b0C, temperature was then kept constant for 8 min. Flame ionisation detector temperature was 310 \u00b0C.\n<\/p><p>Thermo Electron Trace 2000 GC coupled with Thermo Electron DSQ quadrupole mass spectrometer (Thermo Scientific, Rodano, Milan, Italy) with an electron ionisation source at 70 eV in positive mode was applied for the purpose of terpene identification. Some parameters had to be adapted for <a href=\"https:\/\/www.limswiki.org\/index.php\/Gas_chromatography%E2%80%93mass_spectrometry\" title=\"Gas chromatography\u2013mass spectrometry\" class=\"wiki-link\" data-key=\"d7fe02050f81fca3ad7a5845b1879ae2\">gas chromatography\u2013mass spectrometry<\/a> (GC-MS) analysis. Namely, 0.5 \u00b5L of sample was injected with a split ratio of 60:1, flow rate of carrier gas was 1 mL\/min. Other parameters remained the same as for GC-FID analysis. The MS data were acquired in full scan mode from m\/z 50\u2013450 with acquisition frequency of 4.2 scans per second.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"HPLC_analysis\">HPLC analysis<\/span><\/h3>\n<p>Analysis was performed according to Kri\u017eman's published method.<sup id=\"rdp-ebb-cite_ref-Kri.C5.BEmanASimp19_33-5\" class=\"reference\"><a href=\"#cite_note-Kri.C5.BEmanASimp19-33\">[33]<\/a><\/sup> For HPLC analysis, sample solutions (as described below in \"Samples and preparation of extracts\") were further diluted with methanol 100-fold.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Standard_solutions_preparation\">Standard solutions preparation<\/span><\/h3>\n<p>Stock solutions of solid cannabinoid standards (cannabidiol and cannabigerol) were prepared in acetone in the concentration of 2.0 mg\/mL. Working standard solutions of cannabidiol and cannabigerol were prepared in the concentration range of 0.05 to 1.5 mg\/mL, and working standard solutions of cannabichromene, cannabinol, \u03948-tetrahydrocannabinol, and \u03949-tetrahydrocannabinol were prepared in the concentration range of 0.01 to 0.2 mg\/mL. All working solutions contained internal standard of concentration 0.13 mg\/mL and were prepared by diluting stock solution with acetone.\n<\/p><p>Stock solutions of terpene standards were prepared in the concentration of 2.0 mg\/mL in acetone, except for myrcene stock solution, which was prepared in hexane. Working standard solutions consisted of a mix of all terpene standards in the concentration range of 1.0 to 100 \u00b5g\/mL.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Samples_and_preparation_of_extracts\">Samples and preparation of extracts<\/span><\/h3>\n<p>Dried and powdered plant materials (300 mg) were extracted by sonication for 30 min at room temperature with acetone or ethyl acetate containing IS (130 \u00b5g\/mL). Sample solutions were then centrifuged at 16.000g for 10 min and the supernatant was transferred into GC vials.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Quantitation.2C_method_precision.2C_accuracy.2C_sensitivity.2C_linearity.2C_and_stability\">Quantitation, method precision, accuracy, sensitivity, linearity, and stability<\/span><\/h3>\n<p>Injection precision was determined by five injections of working standard solution. Extraction efficiency was assessed by three consecutive extractions of selected plant samples, and then comparing the analyte recovery with the combined recovery of all extraction steps. Accuracy was determined by spiking sample solution with cannabinoid standards at concentration of 0.25 mg\/mL and terpene standards at concentration of 0.025 mg\/mL. Repeatability and intermediate precision were also tested on a homogeneous plant sample. Three replicates were assayed for repeatability, while three replicates were assayed on each of the three consecutive days for intermediate precision.\n<\/p><p>Linearity was checked in the range of standard solutions concentration (0.05 to 1.5 mg\/mL for CBD and CBG; 0.01 to 0.2 mg\/mL for CBC, CBN, \u03948-THC, and \u03949-THC; and 0.001 to 0.1 mg\/mL for terpenes). Correlation coefficients were calculated with intercept values set at zero. Analyte peak areas were normalised by dividing them with IS peak areas. For stability tests, a sample solution was refrigerated at 4 \u00b0C in the dark for 48 h.\n<\/p><p>In order to confirm the correctness of the results, HPLC analysis of properly diluted sample extracts solutions in methanol was performed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>A gas chromatography\u2013flame ionization detection (GC-FID) method for simultaneous analysis of terpenes and cannabinoids in hemp samples has been developed. The main issues concerning the method were ensuring appropriate sample preparation conditions for both terpenes and cannabinoids, successful separation of critical cannabinoid peaks, and quantitative decarboxylation of cannabinoid acidic forms. Acetone proved to be an appropriate solvent for quantitative extraction of all the analytes concerned, which occur in a wide concentration range, using a sufficiently low sample-to-solvent ratio. Separation-wise, the resolution between CBD and CBC peaks was substantially improved by using a relatively polar column with 50% phenyl 50% dimethylpolysiloxane stationary phase. Quantitative decarboxylation was ensured using a high injector temperature and low injection volumes. Peak identity was confirmed by GC-MS. Despite many of the method parameters being near the practical limits in terms of instrumentation capacity like temperature, detector response, etc., the method provides a robust tool for simultaneous quantitative analysis of these two chemically different groups of analytes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>Jure Zeki\u010d is a student of the doctoral program in chemical sciences at the Faculty of Chemistry and Chemical Technology, University of Ljubljana, Ljubljana, Slovenia.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Conceptualization, M.K. and J.Z.; investigation, J.Z.; supervision, M.K.; data analysis, J.Z. and M.K.; writing\u2014original draft preparation, J.Z.; writing\u2014review and editing, J.Z. and M.K. All authors have read and agreed to the published version of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sample_availability\">Sample availability<\/span><\/h3>\n<p>Samples of the compounds are available from the authors.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research was supported by the Slovenian Research Agency (research core funding No. P1-0005).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-AbuharisaMedical18-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbuharisaMedical18_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abuhasira, R.; Shbiro, L.; Landschaft, Y. (2018). \"Medical use of cannabis and cannabinoids containing products - Regulations in Europe and North America\". <i>European Journal of Internal Medicine<\/i> <b>49<\/b>: 2\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ejim.2018.01.001\" target=\"_blank\">10.1016\/j.ejim.2018.01.001<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29329891\" target=\"_blank\">29329891<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+use+of+cannabis+and+cannabinoids+containing+products+-+Regulations+in+Europe+and+North+America&rft.jtitle=European+Journal+of+Internal+Medicine&rft.aulast=Abuhasira%2C+R.%3B+Shbiro%2C+L.%3B+Landschaft%2C+Y.&rft.au=Abuhasira%2C+R.%3B+Shbiro%2C+L.%3B+Landschaft%2C+Y.&rft.date=2018&rft.volume=49&rft.pages=2%E2%80%936&rft_id=info:doi\/10.1016%2Fj.ejim.2018.01.001&rft_id=info:pmid\/29329891&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HutchisonCanna19-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HutchisonCanna19_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hutchison, K.E.; Bidwell, L.C.; Ellingson, J.M. et al. (2019). \"Cannabis and Health Research: Rapid Progress Requires Innovative Research Designs\". <i>Value in Health<\/i> <b>22<\/b> (11): 1289\u201394. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jval.2019.05.005\" target=\"_blank\">10.1016\/j.jval.2019.05.005<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31708066\" target=\"_blank\">31708066<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis+and+Health+Research%3A+Rapid+Progress+Requires+Innovative+Research+Designs&rft.jtitle=Value+in+Health&rft.aulast=Hutchison%2C+K.E.%3B+Bidwell%2C+L.C.%3B+Ellingson%2C+J.M.+et+al.&rft.au=Hutchison%2C+K.E.%3B+Bidwell%2C+L.C.%3B+Ellingson%2C+J.M.+et+al.&rft.date=2019&rft.volume=22&rft.issue=11&rft.pages=1289%E2%80%9394&rft_id=info:doi\/10.1016%2Fj.jval.2019.05.005&rft_id=info:pmid\/31708066&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BeauchetMedical18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BeauchetMedical18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Beauchet, O. (2018). \"Medical cannabis use in older patients: Update on medical knowledge\". <i>Maturitas<\/i> <b>118<\/b>: 56\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.maturitas.2018.10.010\" target=\"_blank\">10.1016\/j.maturitas.2018.10.010<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30415756\" target=\"_blank\">30415756<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+cannabis+use+in+older+patients%3A+Update+on+medical+knowledge&rft.jtitle=Maturitas&rft.aulast=Beauchet%2C+O.&rft.au=Beauchet%2C+O.&rft.date=2018&rft.volume=118&rft.pages=56%E2%80%939&rft_id=info:doi\/10.1016%2Fj.maturitas.2018.10.010&rft_id=info:pmid\/30415756&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EbbertMedical18-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EbbertMedical18_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ebbert, J.O.; Scharf, E.L.; Hurt, R.T. (2018). \"Medical Cannabis\". <i>Mayo Clinic Proceedings<\/i> <b>93<\/b>: 1842\u201347. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.mayocp.2018.09.005\" target=\"_blank\">10.1016\/j.mayocp.2018.09.005<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30522595\" target=\"_blank\">30522595<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+Cannabis&rft.jtitle=Mayo+Clinic+Proceedings&rft.aulast=Ebbert%2C+J.O.%3B+Scharf%2C+E.L.%3B+Hurt%2C+R.T.&rft.au=Ebbert%2C+J.O.%3B+Scharf%2C+E.L.%3B+Hurt%2C+R.T.&rft.date=2018&rft.volume=93&rft.pages=1842%E2%80%9347&rft_id=info:doi\/10.1016%2Fj.mayocp.2018.09.005&rft_id=info:pmid\/30522595&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Blasco-BenitoAppr18-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Blasco-BenitoAppr18_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blasco-Benito, S.; Seijo-Vila, M.; Caro-Villalobos, M. et al. (2018). \"Appraising the \"entourage effect\": Antitumor action of a pure cannabinoid versus a botanical drug preparation in preclinical models of breast cancer\". <i>Biochemical Pharmacology<\/i> <b>157<\/b>: 285\u201393. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.bcp.2018.06.025\" target=\"_blank\">10.1016\/j.bcp.2018.06.025<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29940172\" target=\"_blank\">29940172<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Appraising+the+%22entourage+effect%22%3A+Antitumor+action+of+a+pure+cannabinoid+versus+a+botanical+drug+preparation+in+preclinical+models+of+breast+cancer&rft.jtitle=Biochemical+Pharmacology&rft.aulast=Blasco-Benito%2C+S.%3B+Seijo-Vila%2C+M.%3B+Caro-Villalobos%2C+M.+et+al.&rft.au=Blasco-Benito%2C+S.%3B+Seijo-Vila%2C+M.%3B+Caro-Villalobos%2C+M.+et+al.&rft.date=2018&rft.volume=157&rft.pages=285%E2%80%9393&rft_id=info:doi\/10.1016%2Fj.bcp.2018.06.025&rft_id=info:pmid\/29940172&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BoothTerpenes19-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BoothTerpenes19_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-BoothTerpenes19_6-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Booth, J.K.; Bohlmann, J. (2019). \"Terpenes in Cannabis sativa - From plant genome to humans\". <i>Plant Science<\/i> <b>284<\/b>: 67\u201372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.plantsci.2019.03.022\" target=\"_blank\">10.1016\/j.plantsci.2019.03.022<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31084880\" target=\"_blank\">31084880<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Terpenes+in+Cannabis+sativa+-+From+plant+genome+to+humans&rft.jtitle=Plant+Science&rft.aulast=Booth%2C+J.K.%3B+Bohlmann%2C+J.&rft.au=Booth%2C+J.K.%3B+Bohlmann%2C+J.&rft.date=2019&rft.volume=284&rft.pages=67%E2%80%9372&rft_id=info:doi\/10.1016%2Fj.plantsci.2019.03.022&rft_id=info:pmid\/31084880&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NamdarVaria18-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NamdarVaria18_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-NamdarVaria18_7-1\">7.1<\/a><\/sup> <sup><a href=\"#cite_ref-NamdarVaria18_7-2\">7.2<\/a><\/sup> <sup><a href=\"#cite_ref-NamdarVaria18_7-3\">7.3<\/a><\/sup> <sup><a href=\"#cite_ref-NamdarVaria18_7-4\">7.4<\/a><\/sup> <sup><a href=\"#cite_ref-NamdarVaria18_7-5\">7.5<\/a><\/sup> <sup><a href=\"#cite_ref-NamdarVaria18_7-6\">7.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Namdar, D.; Mazuz, M.; Ion, A. et al. (2018). \"Variation in the compositions of cannabinoid and terpenoids in <i>Cannabis sativa<\/i> derived from inflorescence position along the stem and extraction methods\". <i>Industrial Crops and Products<\/i> <b>113<\/b>: 376\u201382. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.indcrop.2018.01.060\" target=\"_blank\">10.1016\/j.indcrop.2018.01.060<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Variation+in+the+compositions+of+cannabinoid+and+terpenoids+in+%27%27Cannabis+sativa%27%27+derived+from+inflorescence+position+along+the+stem+and+extraction+methods&rft.jtitle=Industrial+Crops+and+Products&rft.aulast=Namdar%2C+D.%3B+Mazuz%2C+M.%3B+Ion%2C+A.+et+al.&rft.au=Namdar%2C+D.%3B+Mazuz%2C+M.%3B+Ion%2C+A.+et+al.&rft.date=2018&rft.volume=113&rft.pages=376%E2%80%9382&rft_id=info:doi\/10.1016%2Fj.indcrop.2018.01.060&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Delgado-PovedanoUntarg20-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Delgado-PovedanoUntarg20_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-Delgado-PovedanoUntarg20_8-1\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-Delgado-PovedanoUntarg20_8-2\">8.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Delgado-Povedano, M.M.; Callado, C.S.-C.; Priego, Capote, F. et al. (2020). \"Untargeted characterization of extracts from <i>Cannabis sativa<\/i> L. cultivars by gas and liquid chromatography coupled to mass spectrometry in high resolution mode\". <i>Talanta<\/i> <b>208<\/b>: 120384. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.talanta.2019.120384\" target=\"_blank\">10.1016\/j.talanta.2019.120384<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Untargeted+characterization+of+extracts+from+%27%27Cannabis+sativa%27%27+L.+cultivars+by+gas+and+liquid+chromatography+coupled+to+mass+spectrometry+in+high+resolution+mode&rft.jtitle=Talanta&rft.aulast=Delgado-Povedano%2C+M.M.%3B+Callado%2C+C.S.-C.%3B+Priego%2C+Capote%2C+F.+et+al.&rft.au=Delgado-Povedano%2C+M.M.%3B+Callado%2C+C.S.-C.%3B+Priego%2C+Capote%2C+F.+et+al.&rft.date=2020&rft.volume=208&rft.pages=120384&rft_id=info:doi\/10.1016%2Fj.talanta.2019.120384&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NuutinenMedic18-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NuutinenMedic18_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-NuutinenMedic18_9-1\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nuutinen, T. (2018). \"Medicinal properties of terpenes found in <i>Cannabis sativa<\/i> and <i>Humulus lupulus<\/i>\". <i>European Journal of Medicinal Chemistry<\/i> <b>157<\/b>: 198\u2013228. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ejmech.2018.07.076\" target=\"_blank\">10.1016\/j.ejmech.2018.07.076<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medicinal+properties+of+terpenes+found+in+%27%27Cannabis+sativa%27%27+and+%27%27Humulus+lupulus%27%27&rft.jtitle=European+Journal+of+Medicinal+Chemistry&rft.aulast=Nuutinen%2C+T.&rft.au=Nuutinen%2C+T.&rft.date=2018&rft.volume=157&rft.pages=198%E2%80%93228&rft_id=info:doi\/10.1016%2Fj.ejmech.2018.07.076&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CittiANovel19-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CittiANovel19_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Citti, C.; Linciano, P.; Russo, F. et al. (2019). \"A novel phytocannabinoid isolated from <i>Cannabis sativa<\/i> L. with an <i>in vivo<\/i> cannabimimetic activity higher than \u03949-tetrahydrocannabinol: \u03949-Tetrahydrocannabiphorol\". <i>Scientific Reports<\/i> <b>9<\/b>: 20335. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41598-019-56785-1\" target=\"_blank\">10.1038\/s41598-019-56785-1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+novel+phytocannabinoid+isolated+from+%27%27Cannabis+sativa%27%27+L.+with+an+%27%27in+vivo%27%27+cannabimimetic+activity+higher+than+%CE%949-tetrahydrocannabinol%3A+%CE%949-Tetrahydrocannabiphorol&rft.jtitle=Scientific+Reports&rft.aulast=Citti%2C+C.%3B+Linciano%2C+P.%3B+Russo%2C+F.+et+al.&rft.au=Citti%2C+C.%3B+Linciano%2C+P.%3B+Russo%2C+F.+et+al.&rft.date=2019&rft.volume=9&rft.pages=20335&rft_id=info:doi\/10.1038%2Fs41598-019-56785-1&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GambaroASimp95-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GambaroASimp95_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-GambaroASimp95_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gambaro, V.E.; Froldi, R.; Saligari, E. et al. (1995). \"A simple method for analyzing cannabis by gas chromatography\". <i>Acta Toxicologica Argentina<\/i> <b>3<\/b>: 11\u201313.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+simple+method+for+analyzing+cannabis+by+gas+chromatography&rft.jtitle=Acta+Toxicologica+Argentina&rft.aulast=Gambaro%2C+V.E.%3B+Froldi%2C+R.%3B+Saligari%2C+E.+et+al.&rft.au=Gambaro%2C+V.E.%3B+Froldi%2C+R.%3B+Saligari%2C+E.+et+al.&rft.date=1995&rft.volume=3&rft.pages=11%E2%80%9313&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GibsonAnal98-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GibsonAnal98_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gibson, C.R.; Williams, R.D.; Browder, R.O. (1998). \"Analysis of Hempen Ale for cannabinoids\". <i>Journal of Analytical Toxicology<\/i> <b>22<\/b> (2): 179. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fjat%2F22.2.179\" target=\"_blank\">10.1093\/jat\/22.2.179<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/9547418\" target=\"_blank\">9547418<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Analysis+of+Hempen+Ale+for+cannabinoids&rft.jtitle=Journal+of+Analytical+Toxicology&rft.aulast=Gibson%2C+C.R.%3B+Williams%2C+R.D.%3B+Browder%2C+R.O.&rft.au=Gibson%2C+C.R.%3B+Williams%2C+R.D.%3B+Browder%2C+R.O.&rft.date=1998&rft.volume=22&rft.issue=2&rft.pages=179&rft_id=info:doi\/10.1093%2Fjat%2F22.2.179&rft_id=info:pmid\/9547418&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RossGC00-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RossGC00_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-RossGC00_13-1\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ross, S.A.; Mehmedic, Z.; Murphy, T.P. et al. (2008). \"GC-MS analysis of the total delta9-THC content of both drug- and fiber-type cannabis seeds\". <i>Journal of Analytical Toxicology<\/i> <b>24<\/b> (8): 715\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fjat%2F24.8.715\" target=\"_blank\">10.1093\/jat\/24.8.715<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11110027\" target=\"_blank\">11110027<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GC-MS+analysis+of+the+total+delta9-THC+content+of+both+drug-+and+fiber-type+cannabis+seeds&rft.jtitle=Journal+of+Analytical+Toxicology&rft.aulast=Ross%2C+S.A.%3B+Mehmedic%2C+Z.%3B+Murphy%2C+T.P.+et+al.&rft.au=Ross%2C+S.A.%3B+Mehmedic%2C+Z.%3B+Murphy%2C+T.P.+et+al.&rft.date=2008&rft.volume=24&rft.issue=8&rft.pages=715%E2%80%937&rft_id=info:doi\/10.1093%2Fjat%2F24.8.715&rft_id=info:pmid\/11110027&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RaharjoMethods04-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RaharjoMethods04_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Raharjo, T.J.; Verpoorte, R. (2004). \"Methods for the analysis of cannabinoids in biological materials: A review\". <i>Phytochemical Analysis<\/i> <b>15<\/b> (2): 79\u201394. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fpca.753\" target=\"_blank\">10.1002\/pca.753<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15116938\" target=\"_blank\">15116938<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Methods+for+the+analysis+of+cannabinoids+in+biological+materials%3A+A+review&rft.jtitle=Phytochemical+Analysis&rft.aulast=Raharjo%2C+T.J.%3B+Verpoorte%2C+R.&rft.au=Raharjo%2C+T.J.%3B+Verpoorte%2C+R.&rft.date=2004&rft.volume=15&rft.issue=2&rft.pages=79%E2%80%9394&rft_id=info:doi\/10.1002%2Fpca.753&rft_id=info:pmid\/15116938&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PellegriniARapid05-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PellegriniARapid05_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pellegrini, M.; Marchei, E.; Pacifici, R. et al. (2005). \"A rapid and simple procedure for the determination of cannabinoids in hemp food products by gas chromatography-mass spectrometry\". <i>Journal of Pharmaceutical and Biomedical Analysis<\/i> <b>36<\/b> (5): 939\u201346. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpba.2004.07.035\" target=\"_blank\">10.1016\/j.jpba.2004.07.035<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+rapid+and+simple+procedure+for+the+determination+of+cannabinoids+in+hemp+food+products+by+gas+chromatography-mass+spectrometry&rft.jtitle=Journal+of+Pharmaceutical+and+Biomedical+Analysis&rft.aulast=Pellegrini%2C+M.%3B+Marchei%2C+E.%3B+Pacifici%2C+R.+et+al.&rft.au=Pellegrini%2C+M.%3B+Marchei%2C+E.%3B+Pacifici%2C+R.+et+al.&rft.date=2005&rft.volume=36&rft.issue=5&rft.pages=939%E2%80%9346&rft_id=info:doi\/10.1016%2Fj.jpba.2004.07.035&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GambaroDeterm02-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GambaroDeterm02_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-GambaroDeterm02_16-1\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-GambaroDeterm02_16-2\">16.2<\/a><\/sup> <sup><a href=\"#cite_ref-GambaroDeterm02_16-3\">16.3<\/a><\/sup> <sup><a href=\"#cite_ref-GambaroDeterm02_16-4\">16.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gambaro, V.; Dell'Acqua, L.; Far\u00e8, F. et al. (2002). \"Determination of primary active constituents in <i>Cannabis<\/i> preparations by high-resolution gas chromatography\/flame ionization detection and high-performance liquid chromatography\/UV detection\". <i>Analtica Chimica Acta<\/i> <b>468<\/b> (2): 245-254. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0003-2670%2802%2900660-8\" target=\"_blank\">10.1016\/S0003-2670(02)00660-8<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determination+of+primary+active+constituents+in+%27%27Cannabis%27%27+preparations+by+high-resolution+gas+chromatography%2Fflame+ionization+detection+and+high-performance+liquid+chromatography%2FUV+detection&rft.jtitle=Analtica+Chimica+Acta&rft.aulast=Gambaro%2C+V.%3B+Dell%27Acqua%2C+L.%3B+Far%C3%A8%2C+F.+et+al.&rft.au=Gambaro%2C+V.%3B+Dell%27Acqua%2C+L.%3B+Far%C3%A8%2C+F.+et+al.&rft.date=2002&rft.volume=468&rft.issue=2&rft.pages=245-254&rft_id=info:doi\/10.1016%2FS0003-2670%2802%2900660-8&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CardeniaDevelop18-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CardeniaDevelop18_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-CardeniaDevelop18_17-1\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-CardeniaDevelop18_17-2\">17.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cardenia, V.; Toschi, T.G.; Scappini, S. et al. (2018). \"Development and validation of a Fast gas chromatography\/mass spectrometry method for the determination of cannabinoids in <i>Cannabis sativa<\/i> L\". <i>Journal of Food and Drug Analysis<\/i> <b>26<\/b> (4): 1283\u201392. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jfda.2018.06.001\" target=\"_blank\">10.1016\/j.jfda.2018.06.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+and+validation+of+a+Fast+gas+chromatography%2Fmass+spectrometry+method+for+the+determination+of+cannabinoids+in+%27%27Cannabis+sativa%27%27+L&rft.jtitle=Journal+of+Food+and+Drug+Analysis&rft.aulast=Cardenia%2C+V.%3B+Toschi%2C+T.G.%3B+Scappini%2C+S.+et+al.&rft.au=Cardenia%2C+V.%3B+Toschi%2C+T.G.%3B+Scappini%2C+S.+et+al.&rft.date=2018&rft.volume=26&rft.issue=4&rft.pages=1283%E2%80%9392&rft_id=info:doi\/10.1016%2Fj.jfda.2018.06.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UNODCRecomm09-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-UNODCRecomm09_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-UNODCRecomm09_18-1\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">United Nations Office on Drug and Crime (2009) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.unodc.org\/documents\/scientific\/ST-NAR-40-Ebook_1.pdf\" target=\"_blank\"><i>Recommended methods for the identification and analysis of cannabis and cannabis products<\/i><\/a>. United Nations. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9789211482423<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.unodc.org\/documents\/scientific\/ST-NAR-40-Ebook_1.pdf\" target=\"_blank\">https:\/\/www.unodc.org\/documents\/scientific\/ST-NAR-40-Ebook_1.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 03 April 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Recommended+methods+for+the+identification+and+analysis+of+cannabis+and+cannabis+products&rft.aulast=United+Nations+Office+on+Drug+and+Crime&rft.au=United+Nations+Office+on+Drug+and+Crime&rft.date=2009&rft.pub=United+Nations&rft.isbn=9789211482423&rft_id=https%3A%2F%2Fwww.unodc.org%2Fdocuments%2Fscientific%2FST-NAR-40-Ebook_1.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeBackerInnov09-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DeBackerInnov09_19-0\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-DeBackerInnov09_19-1\">19.1<\/a><\/sup> <sup><a href=\"#cite_ref-DeBackerInnov09_19-2\">19.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">De Backer, B.; Debrus, B.; Lebrun, P. et al. (2009). \"Innovative development and validation of an HPLC\/DAD method for the qualitative and quantitative determination of major cannabinoids in cannabis plant material\". <i>Journal of Chromatography B<\/i> <b>877<\/b> (32): 4115-4124. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jchromb.2009.11.004\" target=\"_blank\">10.1016\/j.jchromb.2009.11.004<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Innovative+development+and+validation+of+an+HPLC%2FDAD+method+for+the+qualitative+and+quantitative+determination+of+major+cannabinoids+in+cannabis+plant+material&rft.jtitle=Journal+of+Chromatography+B&rft.aulast=De+Backer%2C+B.%3B+Debrus%2C+B.%3B+Lebrun%2C+P.+et+al.&rft.au=De+Backer%2C+B.%3B+Debrus%2C+B.%3B+Lebrun%2C+P.+et+al.&rft.date=2009&rft.volume=877&rft.issue=32&rft.pages=4115-4124&rft_id=info:doi\/10.1016%2Fj.jchromb.2009.11.004&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Aizpurua-OlaizolaIdent14-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Aizpurua-OlaizolaIdent14_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-Aizpurua-OlaizolaIdent14_20-1\">20.1<\/a><\/sup> <sup><a href=\"#cite_ref-Aizpurua-OlaizolaIdent14_20-2\">20.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aizpurua-Olaizola, O.; Omar, J.; Navarro, P. et al. (2014). \"Identification and quantification of cannabinoids in <i>Cannabis sativa<\/i> L. plants by high performance liquid chromatography-mass spectrometry\". <i>Analytical and Bioanalytical Chemistry<\/i> <b>406<\/b>: 7549\u201360. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00216-014-8177-x\" target=\"_blank\">10.1007\/s00216-014-8177-x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identification+and+quantification+of+cannabinoids+in+%27%27Cannabis+sativa%27%27+L.+plants+by+high+performance+liquid+chromatography-mass+spectrometry&rft.jtitle=Analytical+and+Bioanalytical+Chemistry&rft.aulast=Aizpurua-Olaizola%2C+O.%3B+Omar%2C+J.%3B+Navarro%2C+P.+et+al.&rft.au=Aizpurua-Olaizola%2C+O.%3B+Omar%2C+J.%3B+Navarro%2C+P.+et+al.&rft.date=2014&rft.volume=406&rft.pages=7549%E2%80%9360&rft_id=info:doi\/10.1007%2Fs00216-014-8177-x&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZgairDevelop15-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ZgairDevelop15_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-ZgairDevelop15_21-1\">21.1<\/a><\/sup> <sup><a href=\"#cite_ref-ZgairDevelop15_21-2\">21.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zgair, A.; Wong, J.C.M.; Sabri, A. et al. (2015). \"Development of a simple and sensitive HPLC\u2013UV method for the simultaneous determination of cannabidiol and \u03949-tetrahydrocannabinol in rat plasma\". <i>Journal of Pharmaceutical and Biomedical Analysis<\/i> <b>114<\/b>: 145\u201351. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpba.2015.05.019\" target=\"_blank\">10.1016\/j.jpba.2015.05.019<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+of+a+simple+and+sensitive+HPLC%E2%80%93UV+method+for+the+simultaneous+determination+of+cannabidiol+and+%CE%949-tetrahydrocannabinol+in+rat+plasma&rft.jtitle=Journal+of+Pharmaceutical+and+Biomedical+Analysis&rft.aulast=Zgair%2C+A.%3B+Wong%2C+J.C.M.%3B+Sabri%2C+A.+et+al.&rft.au=Zgair%2C+A.%3B+Wong%2C+J.C.M.%3B+Sabri%2C+A.+et+al.&rft.date=2015&rft.volume=114&rft.pages=145%E2%80%9351&rft_id=info:doi\/10.1016%2Fj.jpba.2015.05.019&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GulDeterm15-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GulDeterm15_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-GulDeterm15_22-1\">22.1<\/a><\/sup> <sup><a href=\"#cite_ref-GulDeterm15_22-2\">22.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gul, W.; Gul, S.W.; Radwan, M.M. et al. (2015). \"Determination of 11 Cannabinoids in Biomass and Extracts of Different Varieties of Cannabis Using High-Performance Liquid Chromatography\". <i>Journal of AOAC International<\/i> <b>98<\/b> (6): 1523\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5740%2Fjaoacint.15-095\" target=\"_blank\">10.5740\/jaoacint.15-095<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26651563\" target=\"_blank\">26651563<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determination+of+11+Cannabinoids+in+Biomass+and+Extracts+of+Different+Varieties+of+Cannabis+Using+High-Performance+Liquid+Chromatography&rft.jtitle=Journal+of+AOAC+International&rft.aulast=Gul%2C+W.%3B+Gul%2C+S.W.%3B+Radwan%2C+M.M.+et+al.&rft.au=Gul%2C+W.%3B+Gul%2C+S.W.%3B+Radwan%2C+M.M.+et+al.&rft.date=2015&rft.volume=98&rft.issue=6&rft.pages=1523%E2%80%938&rft_id=info:doi\/10.5740%2Fjaoacint.15-095&rft_id=info:pmid\/26651563&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GieseDevelop15-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GieseDevelop15_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-GieseDevelop15_23-1\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-GieseDevelop15_23-2\">23.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Giese, M.W.; Lewis, M.A.; Giese, L. et al. (2015). \"Development and Validation of a Reliable and Robust Method for the Analysis of Cannabinoids and Terpenes in Cannabis\". <i>Journal of AOAC International<\/i> <b>98<\/b> (6): 1503\u201322. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5740%2Fjaoacint.15-116\" target=\"_blank\">10.5740\/jaoacint.15-116<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26651562\" target=\"_blank\">26651562<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+and+Validation+of+a+Reliable+and+Robust+Method+for+the+Analysis+of+Cannabinoids+and+Terpenes+in+Cannabis&rft.jtitle=Journal+of+AOAC+International&rft.aulast=Giese%2C+M.W.%3B+Lewis%2C+M.A.%3B+Giese%2C+L.+et+al.&rft.au=Giese%2C+M.W.%3B+Lewis%2C+M.A.%3B+Giese%2C+L.+et+al.&rft.date=2015&rft.volume=98&rft.issue=6&rft.pages=1503%E2%80%9322&rft_id=info:doi\/10.5740%2Fjaoacint.15-116&rft_id=info:pmid\/26651562&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CittiMedic16-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CittiMedic16_24-0\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-CittiMedic16_24-1\">24.1<\/a><\/sup> <sup><a href=\"#cite_ref-CittiMedic16_24-2\">24.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Citti, C.; Ciccarella, G.; Braghiroli, D. et al. (2016). \"Medicinal cannabis: Principal cannabinoids concentration and their stability evaluated by a high performance liquid chromatography coupled to diode array and quadrupole time of flight mass spectrometry method\". <i>Journal of Pharmaceutical and Biomedical Analysis<\/i> <b>128<\/b>: 201\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpba.2016.05.033\" target=\"_blank\">10.1016\/j.jpba.2016.05.033<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27268223\" target=\"_blank\">27268223<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medicinal+cannabis%3A+Principal+cannabinoids+concentration+and+their+stability+evaluated+by+a+high+performance+liquid+chromatography+coupled+to+diode+array+and+quadrupole+time+of+flight+mass+spectrometry+method&rft.jtitle=Journal+of+Pharmaceutical+and+Biomedical+Analysis&rft.aulast=Citti%2C+C.%3B+Ciccarella%2C+G.%3B+Braghiroli%2C+D.+et+al.&rft.au=Citti%2C+C.%3B+Ciccarella%2C+G.%3B+Braghiroli%2C+D.+et+al.&rft.date=2016&rft.volume=128&rft.pages=201%E2%80%939&rft_id=info:doi\/10.1016%2Fj.jpba.2016.05.033&rft_id=info:pmid\/27268223&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PatelQual17-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PatelQual17_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-PatelQual17_25-1\">25.1<\/a><\/sup> <sup><a href=\"#cite_ref-PatelQual17_25-2\">25.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Patel, B.; Wene, D.; Fan, Z.T. (2017). \"Qualitative and quantitative measurement of cannabinoids in cannabis using modified HPLC\/DAD method\". <i>Journal of Pharmaceutical and Biomedical Analysis<\/i> <b>146<\/b>: 15\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpba.2017.07.021\" target=\"_blank\">10.1016\/j.jpba.2017.07.021<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Qualitative+and+quantitative+measurement+of+cannabinoids+in+cannabis+using+modified+HPLC%2FDAD+method&rft.jtitle=Journal+of+Pharmaceutical+and+Biomedical+Analysis&rft.aulast=Patel%2C+B.%3B+Wene%2C+D.%3B+Fan%2C+Z.T.&rft.au=Patel%2C+B.%3B+Wene%2C+D.%3B+Fan%2C+Z.T.&rft.date=2017&rft.volume=146&rft.pages=15%E2%80%9323&rft_id=info:doi\/10.1016%2Fj.jpba.2017.07.021&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CiolinoComm18-26\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CiolinoComm18_26-0\">26.0<\/a><\/sup> <sup><a href=\"#cite_ref-CiolinoComm18_26-1\">26.1<\/a><\/sup> <sup><a href=\"#cite_ref-CiolinoComm18_26-2\">26.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ciolino, L.A.; Ranieri, T.L.; Taylor, A.M. (2018). \"Commercial cannabis consumer products part 2: HPLC-DAD quantitative analysis of cannabis cannabinoids\". <i>Forensic Science International<\/i> <b>289<\/b>: 438\u201347. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.forsciint.2018.05.033\" target=\"_blank\">10.1016\/j.forsciint.2018.05.033<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Commercial+cannabis+consumer+products+part+2%3A+HPLC-DAD+quantitative+analysis+of+cannabis+cannabinoids&rft.jtitle=Forensic+Science+International&rft.aulast=Ciolino%2C+L.A.%3B+Ranieri%2C+T.L.%3B+Taylor%2C+A.M.&rft.au=Ciolino%2C+L.A.%3B+Ranieri%2C+T.L.%3B+Taylor%2C+A.M.&rft.date=2018&rft.volume=289&rft.pages=438%E2%80%9347&rft_id=info:doi\/10.1016%2Fj.forsciint.2018.05.033&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MudgeLeaner17-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MudgeLeaner17_27-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-MudgeLeaner17_27-1\">27.1<\/a><\/sup> <sup><a href=\"#cite_ref-MudgeLeaner17_27-2\">27.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mudge, E.M.; Murch, S.J.; Brown, P.N. (2017). \"Leaner and greener analysis of cannabinoids\". <i>Analytical and Bioanalytical Chemistry<\/i> <b>409<\/b>: 3153\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00216-017-0256-3\" target=\"_blank\">10.1007\/s00216-017-0256-3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Leaner+and+greener+analysis+of+cannabinoids&rft.jtitle=Analytical+and+Bioanalytical+Chemistry&rft.aulast=Mudge%2C+E.M.%3B+Murch%2C+S.J.%3B+Brown%2C+P.N.&rft.au=Mudge%2C+E.M.%3B+Murch%2C+S.J.%3B+Brown%2C+P.N.&rft.date=2017&rft.volume=409&rft.pages=3153%E2%80%9363&rft_id=info:doi\/10.1007%2Fs00216-017-0256-3&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangDecarb16-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangDecarb16_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, M.; Wang, Y.-H.; Avula, B. et al. (2016). \"Decarboxylation Study of Acidic Cannabinoids: A Novel Approach Using Ultra-High-Performance Supercritical Fluid Chromatography\/Photodiode Array-Mass Spectrometry\". <i>Cannabis and Cannabinoid Research<\/i> <b>1<\/b> (1): 262-271. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fcan.2016.0020\" target=\"_blank\">10.1089\/can.2016.0020<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Decarboxylation+Study+of+Acidic+Cannabinoids%3A+A+Novel+Approach+Using+Ultra-High-Performance+Supercritical+Fluid+Chromatography%2FPhotodiode+Array-Mass+Spectrometry&rft.jtitle=Cannabis+and+Cannabinoid+Research&rft.aulast=Wang%2C+M.%3B+Wang%2C+Y.-H.%3B+Avula%2C+B.+et+al.&rft.au=Wang%2C+M.%3B+Wang%2C+Y.-H.%3B+Avula%2C+B.+et+al.&rft.date=2016&rft.volume=1&rft.issue=1&rft.pages=262-271&rft_id=info:doi\/10.1089%2Fcan.2016.0020&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DussyIsol05-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DussyIsol05_29-0\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-DussyIsol05_29-1\">29.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dussy, F.E.; Hamberg, C.; Luginb\u00fchl, M. et al. (2005). \"Isolation of \u03949-THCA-A from hemp and analytical aspects concerning the determination of \u03949-THC in cannabis products\". <i>Forensic Science International<\/i> <b>149<\/b> (1): 3\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.forsciint.2004.05.015\" target=\"_blank\">10.1016\/j.forsciint.2004.05.015<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Isolation+of+%CE%949-THCA-A+from+hemp+and+analytical+aspects+concerning+the+determination+of+%CE%949-THC+in+cannabis+products&rft.jtitle=Forensic+Science+International&rft.aulast=Dussy%2C+F.E.%3B+Hamberg%2C+C.%3B+Luginb%C3%BChl%2C+M.+et+al.&rft.au=Dussy%2C+F.E.%3B+Hamberg%2C+C.%3B+Luginb%C3%BChl%2C+M.+et+al.&rft.date=2005&rft.volume=149&rft.issue=1&rft.pages=3%E2%80%9310&rft_id=info:doi\/10.1016%2Fj.forsciint.2004.05.015&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KempCanna95-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KempCanna95_30-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-KempCanna95_30-1\">30.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kemp, P.M.; Abukhalaf, I.K.; Manno, J.E. et al. (2005). \"Cannabinoids in Humans. I. Analysis of \u03949-Tetrahydrocannabinol and Six Metabolites in Plasma and Urine Using GC-MS\". <i>Journal of Analytical Toxicology<\/i> <b>19<\/b> (5): 285\u201391. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fjat%2F19.5.285\" target=\"_blank\">10.1093\/jat\/19.5.285<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabinoids+in+Humans.+I.+Analysis+of+%CE%949-Tetrahydrocannabinol+and+Six+Metabolites+in+Plasma+and+Urine+Using+GC-MS&rft.jtitle=Journal+of+Analytical+Toxicology&rft.aulast=Kemp%2C+P.M.%3B+Abukhalaf%2C+I.K.%3B+Manno%2C+J.E.+et+al.&rft.au=Kemp%2C+P.M.%3B+Abukhalaf%2C+I.K.%3B+Manno%2C+J.E.+et+al.&rft.date=2005&rft.volume=19&rft.issue=5&rft.pages=285%E2%80%9391&rft_id=info:doi\/10.1093%2Fjat%2F19.5.285&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SteinmeyerImprov02-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SteinmeyerImprov02_31-0\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-SteinmeyerImprov02_31-1\">31.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Steinmeyer, S.; Bregel, D.; Warth, S. et al. (2002). \"Improved and validated method for the determination of \u03949-tetrahydrocannabinol (THC), 11-hydroxy-THC and 11-nor-9-carboxy-THC in serum, and in human liver microsomal preparations using gas chromatography\u2013mass spectrometry\". <i>Journal of Chromatography B<\/i> <b>772<\/b> (2): 239\u201348. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS1570-0232%2802%2900102-2\" target=\"_blank\">10.1016\/S1570-0232(02)00102-2<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improved+and+validated+method+for+the+determination+of+%CE%949-tetrahydrocannabinol+%28THC%29%2C+11-hydroxy-THC+and+11-nor-9-carboxy-THC+in+serum%2C+and+in+human+liver+microsomal+preparations+using+gas+chromatography%E2%80%93mass+spectrometry&rft.jtitle=Journal+of+Chromatography+B&rft.aulast=Steinmeyer%2C+S.%3B+Bregel%2C+D.%3B+Warth%2C+S.+et+al.&rft.au=Steinmeyer%2C+S.%3B+Bregel%2C+D.%3B+Warth%2C+S.+et+al.&rft.date=2002&rft.volume=772&rft.issue=2&rft.pages=239%E2%80%9348&rft_id=info:doi\/10.1016%2FS1570-0232%2802%2900102-2&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeghissaDeterm18-32\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LeghissaDeterm18_32-0\">32.0<\/a><\/sup> <sup><a href=\"#cite_ref-LeghissaDeterm18_32-1\">32.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leghissa, A.; Hildenbrand, Z.L.; Foss, F.W. et al. (2018). \"Determination of cannabinoids from a surrogate hops matrix using multiple reaction monitoring gas chromatography with triple quadrupole mass spectrometry\". <i>Journal of Separation Science<\/i> <b>41<\/b> (2): 459\u201368. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fjssc.201700946\" target=\"_blank\">10.1002\/jssc.201700946<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Determination+of+cannabinoids+from+a+surrogate+hops+matrix+using+multiple+reaction+monitoring+gas+chromatography+with+triple+quadrupole+mass+spectrometry&rft.jtitle=Journal+of+Separation+Science&rft.aulast=Leghissa%2C+A.%3B+Hildenbrand%2C+Z.L.%3B+Foss%2C+F.W.+et+al.&rft.au=Leghissa%2C+A.%3B+Hildenbrand%2C+Z.L.%3B+Foss%2C+F.W.+et+al.&rft.date=2018&rft.volume=41&rft.issue=2&rft.pages=459%E2%80%9368&rft_id=info:doi\/10.1002%2Fjssc.201700946&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Kri.C5.BEmanASimp19-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Kri.C5.BEmanASimp19_33-0\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-Kri.C5.BEmanASimp19_33-1\">33.1<\/a><\/sup> <sup><a href=\"#cite_ref-Kri.C5.BEmanASimp19_33-2\">33.2<\/a><\/sup> <sup><a href=\"#cite_ref-Kri.C5.BEmanASimp19_33-3\">33.3<\/a><\/sup> <sup><a href=\"#cite_ref-Kri.C5.BEmanASimp19_33-4\">33.4<\/a><\/sup> <sup><a href=\"#cite_ref-Kri.C5.BEmanASimp19_33-5\">33.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kri\u017eman, M. (2019). \"A simplified approach for isocratic HPLC analysis of cannabinoids by fine tuning chromatographic selectivity\". <i>European Food Research and Technology<\/i> <b>246<\/b>: 315\u201322. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00217-019-03344-7\" target=\"_blank\">10.1007\/s00217-019-03344-7<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+simplified+approach+for+isocratic+HPLC+analysis+of+cannabinoids+by+fine+tuning+chromatographic+selectivity&rft.jtitle=European+Food+Research+and+Technology&rft.aulast=Kri%C5%BEman%2C+M.&rft.au=Kri%C5%BEman%2C+M.&rft.date=2019&rft.volume=246&rft.pages=315%E2%80%9322&rft_id=info:doi\/10.1007%2Fs00217-019-03344-7&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PhenomenexZB-35HT-34\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PhenomenexZB-35HT_34-0\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-PhenomenexZB-35HT_34-1\">34.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Phenomenex. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.phenomenex.com\/Application\/Detail\/22828\" target=\"_blank\">\"Terpenes and Cannabinoids in Marijuana by GC\/FID on ZB-35HT\"<\/a>. <i>Application Search by Name<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.phenomenex.com\/Application\/Detail\/22828\" target=\"_blank\">https:\/\/www.phenomenex.com\/Application\/Detail\/22828<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Terpenes+and+Cannabinoids+in+Marijuana+by+GC%2FFID+on+ZB-35HT&rft.atitle=Application+Search+by+Name&rft.aulast=Phenomenex&rft.au=Phenomenex&rft_id=https%3A%2F%2Fwww.phenomenex.com%2FApplication%2FDetail%2F22828&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RestekRxi-35Sil-35\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RestekRxi-35Sil_35-0\">35.0<\/a><\/sup> <sup><a href=\"#cite_ref-RestekRxi-35Sil_35-1\">35.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Restek. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.restek.com\/chromatogram\/view\/GC_FS0549\/cannabinoids\" target=\"_blank\">\"Medical Cannabis Cannabinoids and Internal Standards on Rxi-35Sil MS\"<\/a>. <i>Restek Searchable Chromatogram Library<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.restek.com\/chromatogram\/view\/GC_FS0549\/cannabinoids\" target=\"_blank\">https:\/\/www.restek.com\/chromatogram\/view\/GC_FS0549\/cannabinoids<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Medical+Cannabis+Cannabinoids+and+Internal+Standards+on+Rxi-35Sil+MS&rft.atitle=Restek+Searchable+Chromatogram+Library&rft.aulast=Restek&rft.au=Restek&rft_id=https%3A%2F%2Fwww.restek.com%2Fchromatogram%2Fview%2FGC_FS0549%2Fcannabinoids&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NissenCharact10-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NissenCharact10_36-0\">36.0<\/a><\/sup> <sup><a href=\"#cite_ref-NissenCharact10_36-1\">36.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nissen, L.; Zatta, A.; Stefanini, I. et al. (2010). \"Characterization and antimicrobial activity of essential oils of industrial hemp varieties (<i>Cannabis sativa<\/i> L.)\". <i>Fitoterapia<\/i> <b>81<\/b> (5): 413\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.fitote.2009.11.010\" target=\"_blank\">10.1016\/j.fitote.2009.11.010<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19969046\" target=\"_blank\">19969046<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Characterization+and+antimicrobial+activity+of+essential+oils+of+industrial+hemp+varieties+%28%27%27Cannabis+sativa%27%27+L.%29&rft.jtitle=Fitoterapia&rft.aulast=Nissen%2C+L.%3B+Zatta%2C+A.%3B+Stefanini%2C+I.+et+al.&rft.au=Nissen%2C+L.%3B+Zatta%2C+A.%3B+Stefanini%2C+I.+et+al.&rft.date=2010&rft.volume=81&rft.issue=5&rft.pages=413%E2%80%939&rft_id=info:doi\/10.1016%2Fj.fitote.2009.11.010&rft_id=info:pmid\/19969046&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RussoTaming11-37\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RussoTaming11_37-0\">37.0<\/a><\/sup> <sup><a href=\"#cite_ref-RussoTaming11_37-1\">37.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Russo, E.B. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\" target=\"_blank\">\"Taming THC: potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\"<\/a>. <i>British Journal of Pharmacology<\/i> <b>163<\/b> (7): 1344\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1476-5381.2011.01238.x\" target=\"_blank\">10.1111\/j.1476-5381.2011.01238.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3165946\/\" target=\"_blank\">PMC3165946<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21749363\" target=\"_blank\">21749363<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Taming+THC%3A+potential+cannabis+synergy+and+phytocannabinoid-terpenoid+entourage+effects&rft.jtitle=British+Journal+of+Pharmacology&rft.aulast=Russo%2C+E.B.&rft.au=Russo%2C+E.B.&rft.date=2011&rft.volume=163&rft.issue=7&rft.pages=1344%E2%80%9364&rft_id=info:doi\/10.1111%2Fj.1476-5381.2011.01238.x&rft_id=info:pmc\/PMC3165946&rft_id=info:pmid\/21749363&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3165946&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OsmanCompar85-38\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OsmanCompar85_38-0\">38.0<\/a><\/sup> <sup><a href=\"#cite_ref-OsmanCompar85_38-1\">38.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Osman, A.; Thorpe, J.W.; Caddy, B. (1985). \"Comparison of Cannabis Samples from Different Origins by the Headspace Technique and an Assessment of Chromatographic Traces Using the r-Matrix\". <i>Journal of the Forensic Science Society<\/i> <b>25<\/b> (6): 427\u201333. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0015-7368%2885%2972431-0\" target=\"_blank\">10.1016\/S0015-7368(85)72431-0<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comparison+of+Cannabis+Samples+from+Different+Origins+by+the+Headspace+Technique+and+an+Assessment+of+Chromatographic+Traces+Using+the+r-Matrix&rft.jtitle=Journal+of+the+Forensic+Science+Society&rft.aulast=Osman%2C+A.%3B+Thorpe%2C+J.W.%3B+Caddy%2C+B.&rft.au=Osman%2C+A.%3B+Thorpe%2C+J.W.%3B+Caddy%2C+B.&rft.date=1985&rft.volume=25&rft.issue=6&rft.pages=427%E2%80%9333&rft_id=info:doi\/10.1016%2FS0015-7368%2885%2972431-0&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FranchinaInDepth20-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FranchinaInDepth20_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Franchina, F.A.; Dubois, L.M.; Focant, J.-F. (2020). \"In-Depth Cannabis Multiclass Metabolite Profiling Using Sorptive Extraction and Multidimensional Gas Chromatography with Low- and High-Resolution Mass Spectrometry\". <i>Analytical Chemistry<\/i> <b>92<\/b> (15): 10512-10520. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Facs.analchem.0c01301\" target=\"_blank\">10.1021\/acs.analchem.0c01301<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32602704\" target=\"_blank\">32602704<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=In-Depth+Cannabis+Multiclass+Metabolite+Profiling+Using+Sorptive+Extraction+and+Multidimensional+Gas+Chromatography+with+Low-+and+High-Resolution+Mass+Spectrometry&rft.jtitle=Analytical+Chemistry&rft.aulast=Franchina%2C+F.A.%3B+Dubois%2C+L.M.%3B+Focant%2C+J.-F.&rft.au=Franchina%2C+F.A.%3B+Dubois%2C+L.M.%3B+Focant%2C+J.-F.&rft.date=2020&rft.volume=92&rft.issue=15&rft.pages=10512-10520&rft_id=info:doi\/10.1021%2Facs.analchem.0c01301&rft_id=info:pmid\/32602704&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AdamsChemist04-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AdamsChemist04_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Adams, D.J.; Dyson, P.J.; Tavener, S.J. (2004). <i>Chemistry in Alternative Reaction Media<\/i>. Wiley. p. 4. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 0471498491.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Chemistry+in+Alternative+Reaction+Media&rft.aulast=Adams%2C+D.J.%3B+Dyson%2C+P.J.%3B+Tavener%2C+S.J.&rft.au=Adams%2C+D.J.%3B+Dyson%2C+P.J.%3B+Tavener%2C+S.J.&rft.date=2004&rft.pages=p.%26nbsp%3B4&rft.pub=Wiley&rft.isbn=0471498491&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RestekRxi-624Sil-41\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RestekRxi-624Sil_41-0\">41.0<\/a><\/sup> <sup><a href=\"#cite_ref-RestekRxi-624Sil_41-1\">41.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Restek. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.restek.com\/chromatogram\/view\/GC_FS0515\" target=\"_blank\">\"Terpenes in Blueberry Jack Medical Cannabis on Rxi-624Sil MS\"<\/a>. <i>Restek Searchable Chromatogram Library<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.restek.com\/chromatogram\/view\/GC_FS0515\" target=\"_blank\">https:\/\/www.restek.com\/chromatogram\/view\/GC_FS0515<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 April 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Terpenes+in+Blueberry+Jack+Medical+Cannabis+on+Rxi-624Sil+MS&rft.atitle=Restek+Searchable+Chromatogram+Library&rft.aulast=Restek&rft.au=Restek&rft_id=https%3A%2F%2Fwww.restek.com%2Fchromatogram%2Fview%2FGC_FS0515&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RestekRxi-624SilStand1_2-42\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RestekRxi-624SilStand1_2_42-0\">42.0<\/a><\/sup> <sup><a href=\"#cite_ref-RestekRxi-624SilStand1_2_42-1\">42.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Restek. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.restek.com\/chromatogram\/view\/GC_GN1200\" target=\"_blank\">\"Cannabis Terpenes Standard #1 & #2 on Rxi-624Sil MS\"<\/a>. <i>Restek Searchable Chromatogram Library<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.restek.com\/chromatogram\/view\/GC_GN1200\" target=\"_blank\">https:\/\/www.restek.com\/chromatogram\/view\/GC_GN1200<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 April 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cannabis+Terpenes+Standard+%231+%26+%232+on+Rxi-624Sil+MS&rft.atitle=Restek+Searchable+Chromatogram+Library&rft.aulast=Restek&rft.au=Restek&rft_id=https%3A%2F%2Fwww.restek.com%2Fchromatogram%2Fview%2FGC_GN1200&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FausettAnal20-43\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FausettAnal20_43-0\">43.0<\/a><\/sup> <sup><a href=\"#cite_ref-FausettAnal20_43-1\">43.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Fausett, A. (26 May 2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.agilent.com\/cs\/library\/applications\/application-terpenes-8890-gcms-headspace-5994-1497en-agilent.pdf\" target=\"_blank\">\"Analysis of Terpene and Terpenoid Content in Cannabis Sativa Using Headspace with GC\/MSD\"<\/a> (PDF). Agilent Technologies, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.agilent.com\/cs\/library\/applications\/application-terpenes-8890-gcms-headspace-5994-1497en-agilent.pdf\" target=\"_blank\">https:\/\/www.agilent.com\/cs\/library\/applications\/application-terpenes-8890-gcms-headspace-5994-1497en-agilent.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 April 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Analysis+of+Terpene+and+Terpenoid+Content+in+Cannabis+Sativa+Using+Headspace+with+GC%2FMSD&rft.atitle=&rft.aulast=Fausett%2C+A.&rft.au=Fausett%2C+A.&rft.date=26+May+2020&rft.pub=Agilent+Technologies%2C+Inc&rft_id=https%3A%2F%2Fwww.agilent.com%2Fcs%2Flibrary%2Fapplications%2Fapplication-terpenes-8890-gcms-headspace-5994-1497en-agilent.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194057\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.806 seconds\nReal time usage: 1.781 seconds\nPreprocessor visited node count: 33108\/1000000\nPreprocessor generated node count: 42035\/1000000\nPost\u2010expand include size: 248969\/2097152 bytes\nTemplate argument size: 83206\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 686.314 1 - -total\n 85.63% 587.703 1 - Template:Reflist\n 71.51% 490.751 43 - Template:Citation\/core\n 63.75% 437.498 36 - Template:Cite_journal\n 7.68% 52.690 5 - Template:Cite_web\n 6.86% 47.093 53 - Template:Citation\/identifier\n 5.42% 37.215 1 - Template:Infobox_journal_article\n 5.19% 35.645 1 - Template:Infobox\n 3.63% 24.926 47 - Template:Citation\/make_link\n 3.59% 24.608 2 - Template:Cite_book\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12292-0!*!0!!en!5!* and timestamp 20210429194055 and revision id 41276\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp\">https:\/\/www.limswiki.org\/index.php\/Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","f48cdd8c7fc25a130c45d85871f3ab72_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3b\/Fig1_Zeki%C4%8D_Molecules2020_25-24.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b9\/Fig2_Zeki%C4%8D_Molecules2020_25-24.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/da\/Fig3_Zeki%C4%8D_Molecules2020_25-24.png"],"f48cdd8c7fc25a130c45d85871f3ab72_timestamp":1619725255,"a40f37542af50527bcd32bf00da69900_type":"article","a40f37542af50527bcd32bf00da69900_title":"Accounting and the US cannabis industry: Federal financial regulations and the perspectives of certified public accountants and cannabis businesses owners (Owens-Ott 2020)","a40f37542af50527bcd32bf00da69900_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners","a40f37542af50527bcd32bf00da69900_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Accounting and the US cannabis industry: Federal financial regulations and the perspectives of certified public accountants and cannabis businesses owners\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nAccounting and the US cannabis industry: Federal financial regulations and the perspectives\r\nof certified public accountants and cannabis businesses ownersJournal\n \nJournal of Cannabis ResearchAuthor(s)\n \nOwens-Ott, G. SuzanneAuthor affiliation(s)\n \nColorado Mesa UniversityPrimary contact\n \nEmail: Send online messageYear published\n \n2020Volume and issue\n \n2Article #\n \n41DOI\n \n10.1186\/s42238-020-00049-7ISSN\n \n2522-5782Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/jcannabisresearch.biomedcentral.com\/articles\/10.1186\/s42238-020-00049-7Download\n \nhttps:\/\/jcannabisresearch.biomedcentral.com\/track\/pdf\/10.1186\/s42238-020-00049-7.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n\n2.1 Accounting and internal control \n2.2 CPA requirements and risks \n2.3 Purpose of this study \n\n\n3 Methods \n\n3.1 Participants \n3.2 Data collection \n3.3 Data analysis \n\n\n4 Results \n\n4.1 Research question 1: Why are some CPAs unwilling to service the cannabis industry? \n4.2 Research question 2: How do CRBs compensate for lack of CPA services? \n4.3 Research question 3: What does a CPA need to know prior to engaging to service the industry? \n\n\n5 Discussion \n\n5.1 Limitations \n5.2 Recommendations for future research \n\n\n6 Conclusion \n7 Abbreviations, acronyms, and initialisms \n8 Supplementary information \n9 Acknowledgements \n\n9.1 Author contributions \n9.2 Ethics approval and consent to participate \n9.3 Funding \n9.4 Availability of data and materials \n9.5 Conflict of interest \n\n\n10 References \n11 Notes \n\n\n\nAbstract \nBackground: Cannabis-related businesses (CRBs), in states where cannabis is legal, may be unable to obtain professional financial services such as banking, insurance, and accounting because of federal laws and regulations. This qualitative study investigated the following research questions.\n\n1. Why are some certified public accountants (CPAs) unwilling to provide services to cannabis-related businesses?\n2. How do CRBs compensate for lack of CPA services?\n3. What does a CPA need to know about the cannabis industry prior to engaging to provide services to CRBs?\nMethods: Data for this grounded-theory qualitative study was gathered from twenty-three semi-structured phone and face-to-face interviews. Ten cannabis-related business owners were recruited from a convenience sample after attempting a broad recruiting effort. Thirteen CPAs with active licenses in Colorado or Washington State participated from firms of varying size and willingness to serve the cannabis industry. The individual interviews, which lasted from twenty minutes to more than an hour, focused on the participants\u2019 perceptions of the complexities of accounting and tax compliance for cannabis businesses.\nResults: Eight of the thirteen CPAs interviewed would not provide services to the cannabis industry with the primary reason given that cannabis is federally illegal. All ten of the cannabis business owners interviewed indicated they engage a CPA to provide tax services. Seven out of ten CRB participants and ten of the thirteen CPA participants indicated that extensive industry knowledge is needed for an accountant to competently provide services to a CRB.\nConclusions: CRB owners need to carefully consider the industry knowledge and experience of a potential CPA prior to engaging them. This study shows that U.S. CPAs should weigh the risk of federal prosecution and potential loss of the CPA license when deciding whether to serve a CRB client. The study also found that a CPA must commit to acquiring and maintaining substantial specialized knowledge related to tax Code Section 280E, internal controls for a cash-only or cash-intensive business, and the workings of the cannabis industry under the current regulatory conditions.\nKeywords: cannabis, marijuana, accounting, business, certified public accountants, financial regulation, Colorado, Washington State\n\nBackground \nThere are a growing number of cannabis-related businesses (CRBs) in the United States, as many states have legalized cannabis for medical and\/or recreational use even though it remains illegal at the federal level under the Controlled Substances Act (CSA). Thirty-three states plus the District of Columbia, Puerto Rico, the U.S. Virgin Islands, and Guam have legalized medical cannabis, while eleven states plus the District of Columbia and the Northern Mariana Islands have also legalized recreational cannabis usage as of September 2020.[1] Table 1 shows the legalized states by category. Cannabis has been listed as a Schedule I drug since 1970.[2] Regardless of state laws, cannabis is a Schedule I drug according to the federal government, and those found trafficking in cannabis could face criminal prosecution.[3] The Rohrabacher-Blumenauer amendment prohibits the use of federal funds to prosecute a cannabis-related business activity that operates within \u201cStates that have legalized the use of medical marijuana,\u201d and was most recently extended to the 2019\u20132020 fiscal year in the 2020 Consolidated Appropriations Act.[4]\n\n\n\n\n\n\n\nTable 1. States in which cannabis is legal for medical and\/or recreational use. Abbreviations are standard US Postal Service abbreviations.[1] CBD = cannabidiol; THC =tetrahydrocannabinol.\n\n\nStates in which cannabis is legal only for medical use\n\nStates in which cannabis is legal for medical and recreational use\n\nStates in which only CBD (low THC) cannabis is allowed\n\nStates in which cannabis remains illegal\n\n\nAR, AZ, CT, DC, DE, FL, HI, LA, MD, MN, MO, MT, ND, NH, NJ, NM, NY, OH, OK, PA, RI, UT, and WV\n\nAK, CA, CO, IL, MA, ME, MI, NV, OR, VT, and WA\n\nAL, GA, IA, IN, KY, MS, NC, SC, TN, TX, VA, WI, and WY\n\nID, KS, NE, SD\n\n\n\nCannabis and hemp are both varieties of the cannabis plant but with different tetrahydrocannabinol (THC) content. Hemp products became federally legal with the passage of the [[wikipedia:[Agriculture Improvement Act of 2018|Agriculture Improvement Act of 2018]].[5] The bill \u201cremoved hemp, defined as cannabis (Cannabis sativa L.) and derivatives of cannabis with extremely low concentrations of the psychoactive compound delta-9-tetrahydrocannabinol (THC) (no more than 0.3 percent THC on a dry weight basis), from the definition of marijuana in the Controlled Substances Act (CSA).\u201d[6] Cannabis remains federally illegal and is the topic of this study.\nThis conflict between federal and state laws is problematic because cannabis-related businesses and ancillary businesses operating in legalized states and complying with all state laws are unable to fully comply with federal laws. Many cannabis businesses in states where cannabis is legal are facing difficulties in obtaining professional financial services because cannabis is still a controlled substance at the federal level.[7] For example, CRBs face difficulty obtaining banking services such as checking, credit cards, electronic transfers, and loans, which results in cash-only or cash-intensive business operation.[7] As such, many certified public accountants (CPAs) may be unwilling to provide their accounting and tax services to CRBs due to increased risks associated with the industry.[7][8][9]\nBecause cannabis is a Schedule I drug, a cannabis business is subject to Internal Revenue Service (IRS) Code Section 280E, which disallows the deduction of ordinary business expenses in arriving at taxable income.[10] As such, Code Section 280E results in significantly higher effective tax rates for cannabis businesses than for other businesses.[11][12] The Racketeer Influenced and Corrupt Organizations Act (RICO) of 1970 provides for federal criminal prosecution of individuals involved with criminal activities, including drug trafficking.[13] One does not have to be directly involved with the criminal enterprise to be prosecuted under RICO, as it applies to others associated with the criminal enterprise, include those providing professional services such as accountants, lawyers, and bankers.[14][15]\nCannabis businesses are also greatly affected by the Bank Secrecy Act (BSA) and anti-money laundering regulations that are designed to help identify and report money laundering activity.[16] The Financial Crimes Enforcement Network (FinCEN), a bureau of the U.S. Department of the Treasury, exists to \u201csafeguard the financial system from illicit use and combat money laundering.\u201d[17] In practical terms, FinCEN guidance stated that providing banking services to cannabis businesses was illegal and required banks to self-report this federally illegal activity.[18][19] Due to the BSA risks and FinCEN reporting requirements, most large banks will not provide accounts for cannabis-related businesses.[20] However, there is evidence that some banks and credit unions are serving the industry. According to FinCEN, as of June 2020, 695 financial institutions in the U.S. are reportedly serving the cannabis industry; this is down from a high of 747 institutions in November 2019.[21]\nBills have been proposed that would change the federal landscape for the cannabis industry. For example, H.R. 1588, the Ending Federal Marijuana Prohibition Act of 2019[22], was introduced in the House of Representatives in March of 2019 and referred to the House Energy and Commerce and House Judiciary committees. There have been several attempts to pass legislation to ease the banking burden for the industry, including the Secure and Fair Enforcement (SAFE) Banking Act of 2019, which passed in the House of Representatives but has not yet passed the Senate.[23] There have been numerous cannabis-related bills introduced in the 116th Congress in 2019\u20132020, ranging from the MAPLE Act, which would remove cannabis from the list of crimes that would prevent an immigrant entrance into the U.S., to the MORE Act, which provides for the total decriminalization of cannabis and expungement of cannabis-related convictions. However, as of October 2020, none of these bills have passed both chambers.[24][25] Changes in federal legislation could potentially reduce, if not eliminate, the business problem studied herein. For example, if federal prohibition were eliminated, cannabis clients would be no different than any other client in a legal industry from the perspective of a CPA. Passage of a cannabis banking act would allow the industry to utilize regular banking services and eliminate some risk associated with a cash-intensive type of business.\n\nAccounting and internal control \nThere are several unique accounting issues that affect the cannabis industry. Due to the state and local regulatory requirements, CRBs need to maintain effective accounting records. Tax Code Section 280E requires attention to inventory accounting while following the tax requirements.[26] Lack of banking availability requires special attention to internal controls to safeguard cash and maintain good accounting records of cash transactions. State regulations also require thorough accounting records as well as strong internal controls.[27][28][29]\nInternal controls are also important to a cannabis business. These are the activities that an organization takes to provide reasonable assurance that the company can meet its goals.[30][31] Failure to comply with state, local, or tax regulations can result in penalties, fines, and potential revocation of the cannabis license.[32] For example, effective internal controls were missing in the Alterman v. Commissioner case in which a medical cannabis company failed to maintain adequate records and failed to understand how to comply with Tax Code Section 280E.[33] Due to complexities in accounting and regulation, cannabis businesses should be perpetually ready for a potential audit by regulators or the IRS.[32] CPAs can provide small businesses, including cannabis businesses, with guidance to develop and implement effective internal controls.[34]\nA cash-intensive business requires some unique internal controls to safeguard the asset and to maintain adequate records of cash transactions.[30] The IRS defines a cash-intensive business as one that receives most of its revenues in cash and\/or pays many of its expenses in cash.[35] Cash may be stolen by employees and is most susceptible to theft when entering or exiting the business.[30] To maintain proper accounting records to support potential tax and other regulatory audits, a cash-intensive business should \u201cdocument the flow of each receipt or revenue from the customer\u2019s hands to the business, to the final end in the business bank account or as payment for a business expense.\u201d[36]\nPhysical controls may be one of the most crucial internal controls, and security must be considered for areas \u201cto include property, office buildings, warehouses, utility rooms \u2026 and vehicles as well as employees, contractors, and visitors.\u201d[37] Physical controls over cash and cannabis inventory include keeping them in a safe or vault, video surveillance, and promptly depositing cash into a bank account if the cannabis business has one.[28][38][39] Physical controls should be tested frequently to ensure continuous safeguarding of assets.[37] Frequent physical inventory counts are needed, and differences between the actual count and inventory records should be examined right away.[34] Colorado, for examples, has regulations that require daily reconciliation of inventory on-hand to the inventory tracking system.[27][28]\n\nCPA requirements and risks \nCPAs must adhere to an extremely high standard of ethical conduct as provided by the American Institute of Certified Public Accountants (AICPA) Code of Professional Conduct, as well as any individual state codes of conduct.[40] Unlike many business professionals, CPAs obtain a license or permit to practice from the state(s) in which they provide services.[8] A violation of ethics or \u201clack of good moral character\u201d could potentially result in a loss of the CPA license, and therefore the potential loss of the ability to earn a living.[8]\nMost state boards of accountancy have provided somewhat unclear official guidance on the provision of services for the cannabis industry.[8] The Washington State Board of Accountancy (BOA) stated in 2014, and again in 2018, that a CPA\u2019s provision of services to a cannabis-related business does not constitute a violation of the BOA\u2019s rules. This statement followed the March 2018 signing of the Engrossed Substitute Senate Bill 5928, which states that the provision of services to a CRB by a CPA does not, by itself, constitute a crime.[41] The Washington BOA further recommended that CPAs consider the risks associated with serving the cannabis industry and that CPAs engage an attorney for counsel.[41] As for Colorado, their BOA issued a position statement on December 16, 2015 that indicated the provision of services by a Certified Public Accountant to a cannabis business is not \u201cspecifically prohibited by the Accountancy Act.\u201d[42] The Colorado Board went on to caution CPAs that their position statement does not constitute an endorsement for CPAs to enter the industry.[42]\nIn January of 2019, the National Association of State Boards of Accountancy (NASBA), in conjunction with the AICPA, published a document entitled \"Providing services to businesses in the marijuana industry: A sample of current board positions.\"[42] This paper summarized the board positions for Alaska, Arizona, Arkansas, Colorado, Connecticut, Florida, Iowa, Maryland, Massachusetts, Michigan, New Mexico, Nevada, Oregon, and Washington. This sample includes states in which cannabis is legal only for medical purposes, states in which cannabis is legal for recreational or medical usage, and a state in which cannabis remains prohibited. All state boards in the report, except New Mexico, indicated that the CPA would not face disciplinary action by the board for providing services to a cannabis business, assuming the CPA was in compliance with all state laws.[42] This was even the case for Iowa, where only low-THC, high-CBD (cannabidiol) cannabis is allowed; the Iowa BOA indicated that Iowa CPAs serving cannabis businesses in legalized states would not face disciplinary action.[42]\nWhile providing services to cannabis businesses does not by itself necessarily constitute a violation of good moral character, other problems can result from a CPA serving the cannabis industry.[43] In some cases, a CPA may be found to have \u201caided and abetted\u201d or been involved in a \u201cconspiracy to violate\u201d the federal CSA or racketeering laws.[43] A CPA may also be exposed to criminal investigation and\/or prosecution as well as potential fines, penalties, and sanctions if he participates in \u201cdishonest, fraudulent, or criminal acts\u201d associated with the cannabis industry.[43]\nCannabis businesses have a higher likelihood of income tax audit than other businesses due to complexities of Tax Code Section 280E.[44] The large number of cash transactions may result in the absence of a clear paper trail, which increases the risk of tax evasion and makes cannabis businesses targets for federal tax audits.[16] Some state cannabis regulations\u2014including those found in New Mexico, Minnesota, and Colorado\u2014may require a financial statement audit of a cannabis business to be completed by a CPA.[8][28]\nAdditionally, CPAs may believe associating with the cannabis industry could damage their reputation in the business community with current or prospective clients. Devers et al. have shown that reputation of a firm can indeed be seen by outsiders as an indication of the firm\u2019s quality of services.[45] CPA firms may also be viewed as less than legitimate based on their overall association with the somewhat controversial cannabis industry.[45] Core-stigmatized organizations are those for whom outsiders have a \u201cperceived violation of social norms\u201d and may be looked at unfavorably.[46] Current or prospective clients may avoid associating with a CPA firm who works in the cannabis industry because they worry that negative stigma may transfer to them.[47] However, some CPAs may determine that they are willing to accept such core stigma as part of their business strategy, as may be the case for CPA firms who specialize in cannabis clients.[46]\nLastly, professional standards require that a CPA only take on engagements for which he or she has the appropriate technical knowledge about the industry to complete the work competently.[42] The complexities of the cannabis industry make meeting this standard costly and not necessarily worth the risks to many CPAs.[8] In general, CPAs need to evaluate the many risks associated with providing services to the industry and may determine that the risks outweigh the benefits and choose to not take on cannabis clients.[48]\n\nPurpose of this study \nGiven the previously discussed, this qualitative study investigated the following research questions:\n\n1. Why are some certified public accountants (CPAs) unwilling to provide services to cannabis-related businesses?\n2. How do CRBs compensate for lack of CPA services?\n3. What does a CPA need to know about the cannabis industry prior to engaging to provide services to CRBs?\nMethods \nQualitative research is particularly well-suited for the \u201cearly stages of research\u201d as in the case of a young industry such as cannabis.[49] The grounded theory approach to qualitative research utilizes \u201csystematical methodological procedures\u201d to identify theories as they \u201cemerge from the data.\u201d[50] Grounded theory is particularly appropriate for research such as this study, which seeks to generate theory based upon the reported data from participants.[51] Grounded theory requires one to continuously compare and analyze data previously collected to the new data being collected to identify patterns and themes, and produce theories. Interviews of participants are often used as a method of data collection in grounded theory research.[50][51][52]\nSemi-structured interviews of cannabis business owners were used to learn about the financial difficulties CRBs face, particularly related to accounting and finding CPA services. Semi-structured interviews of CPAs provided an understanding of the willingness or hesitancy to provide services to CRBs, the potential risks related to the industry, and accounting and tax issues relevant to CRBs. The interview questions for CPAs may be found in the supplementary information as Additional file 1: Appendix A, and the questions for CRB owners as Additional file 2: Appendix B. Institutional review board (IRB) approval was obtained from California Southern University prior to data collection. Signed statements of informed consent from all participants were emailed or hand delivered during face-to-face interviews. Interviews were one-on-one. Nineteen interviews were conducted via telephone and four conducted in-person between February 2019 and May 2019. Interviews lasted from approximately 20 minutes to more than an hour.\n\nParticipants \nA list of 3,076 active CPA firms was downloaded from the Colorado and Washington State Board of Accountancy websites on December 2 and 3, 2018, respectively. The listed firms\u2019 websites were then reviewed to locate contact information for partners and managers, who were then emailed a recruitment letter. This was a highly manual process and not all websites had contact information for their employees posted, resulting in less than half the number of email contacts compared to the number of firms. While 1,249 requests for participation were emailed, only four willing participants responded. A convenience sample was recruited for the remaining CPA participants by posting requests for participation on LinkedIn and Facebook and through conversations and emails with accounting colleagues in both states. Deliberate effort was made to include participants from a variety of sizes of firms as well as from firms who did serve the cannabis industry and firms that did not. No participants withdrew during the interviews; however, one CPA declined the interview and simply provided a statement that his firm does not provide services to the cannabis industry because it remains federally illegal.\nColorado and Washington State both maintain publicly available listings of licensed cannabis businesses. The Colorado Department of Revenue publishes lists of licensed cannabis facilities on their state website. The listings reported approximately 2,200 licensed Colorado retail stores, retail manufacturers, retail cultivators, and medical cultivators for December 2018. These lists contained only company name, city, zip code, and license number. Seventy randomly selected companies from the combined lists were researched on the internet to locate email or phone information. A request for participation was emailed if an email address was listed or if there was a \u201ccontact us\u201d option on the website.\nThe Washington State Liquor and Cannabis Board publishes a report of monthly \u201cSales Activity by License Number.\u201d For December 2018, 433 cannabis licenses reported sales activity. The 433 license numbers were then cross referenced to the \u201cWashington Listing of Marijuana Applicants\u201d report which provides company name, address, and phone number. Since phone numbers were readily available on this report, phone calls were made to request participation from randomly selected licensees throughout the list.\nForty-five licensed cannabis businesses were called and seventy were emailed to request owner participation, but finding CRB participants proved to be extremely difficult. No emails were answered. Phone calls never got past the employee answering the phone, meaning the employee indicated they would not be interested or would leave a message for the owner to respond. Only two CRB owners responded, but they both declined participation and generally expressed unease in sharing any information related to their business with a stranger. It became apparent that obtaining participation in this manner was not effective.\nNext, requests for participation were posted on LinkedIn and Facebook and requested through conversations with colleagues to identify and recruit potential CRB owners in both states. This led to five leads where there was a personal relationship between someone in our professional network and a CRB owner. One of those five, a small Colorado retailer, did not agree to participate as they were uncomfortable sharing their business information. The other four CRB owners agreed to participate and provided additional names of CRB owners in their networks to contact at the conclusion of their interviews. This process of referrals continued until the study reached ten CRB participants. No participants withdrew during the interviews. The unresponsive CRBs varied in terms of location and type of firm, as did those CRBs that participated. Tables 2 and 3 show the descriptive data for the participants.\n\n\n\n\n\n\n\nTable 2. Overview of CRBs participating in interviews.\r\n \r\naFirst two digits of participant ID represent the state, next three digits indicate the participant was a CRB owner, and last digit represents the number of the interviewee.\n\n\nParticipant IDa\n\nType of cannabis business\n\nYear established\n\nLegal structure\n\nApproximate 2018 revenues\n\n\nWACRB1\n\nProducer\/Processor\n\n2014\n\nLLC\n\n$3.5M\n\n\nWACRB2\n\nProducer\/Processor\n\n2015\n\nLLC\n\n$600k\n\n\nWACRB3\n\nProducer\/Processor\n\n2015\n\nPartnership\n\n$5k (closing)\n\n\nWACRB4\n\nProcessor\n\n2013\n\nLLC\n\nDeclined\n\n\nWACRB5\n\nRetailer\n\n2017\n\nPartnership\n\n$5.6M\n\n\nCOCRB1\n\nRetailer\n\n2014\n\nLLC\n\n$2.5M\n\n\nCOCRB2\n\nRetailer\n\n2014\n\nLLC\n\n$4.0M\n\n\nCOCRB3\n\nProducer\/Processor\n\n2014\n\nS-Corp\n\n$5.0 to 6.0M\n\n\nCOCRB4\n\nProducer\n\n2018\n\nS-Corp\n\n$0 (start-up)\n\n\nCOCRB5\n\nRetailer\n\n2009 (medical)\n\nLLC\n\n$1.0M\n\n\n\n\n\n\n\n\n\nTable 3. Overview of CPAs participating in interviews.\r\n \r\naFirst two digits of participant ID represent the state, second three digits indicate the participant was a Certified Public Accountant, and last digit represents the number of the interviewee.\r\nbChoices were: single CPA; local CPA firm with multiple CPAs; regional CPA firm; national CPA firm; Big 4 CPA firm; or not in public accounting.\r\ncCOCPA11 was a partner at a Big 4 accounting firm that declined to do a phone interview but responded via email that the firm\u2019s policy is to not serve cannabis businesses because they are illegal at the federal level.\n\n\nParticipant IDa\n\nPartner or manager within firm\n\nPrimary county or US Zip code of practice\n\nType of firmb\n\nDo you or your firm provide services to CRBs?\n\n\nWACPA1\n\nPartner\n\n98501\n\nLocal\n\nYes\n\n\nWACPA2\n\nPartner\n\nVarious\n\nSingle CPA\n\nYes\n\n\nWACPA3\n\nPartner\n\n98004\n\nLocal\n\nYes\n\n\nCOCPA1\n\nPartner\n\n81501\n\nSingle CPA\n\nNo\n\n\nCOCPA2\n\nPartner\n\n80237\n\nNational\n\nNo\n\n\nCOCPA3\n\nPartner\n\nNationwide\n\nNational\n\nYes\n\n\nCOCPA4\n\nManager\n\n81501\n\nNational\n\nNo\n\n\nCOCPA5\n\nPartner\n\n81501\n\nRegional\n\nNo\n\n\nCOCPA6\n\nPartner\n\n80433\n\nLocal\n\nNo\n\n\nCOCPA7\n\nPartner\n\n80433 and 81501\n\nLocal\n\nNo\n\n\nCOCPA8\n\nPartner\n\n80901\n\nSingle CPA\n\nYes\n\n\nCOCPA9\n\nPartner\n\nDenver\n\nRegional\n\nNo\n\n\nCOCPA10\n\nManager\n\n81521\n\nLocal\n\nNo\n\n\nCOCPA11\n\nPartner\n\nNationwide\n\nBig 4\n\nNo\n\n\n\nData collection \nThree central research questions were the focus of the study, with sub-questions asked to narrow the focus as the interview was conducted. As interviews took place, responses were analyzed for clarity and understanding. At times, probes were asked to \u201cfollow up something already asked\u201d and to seek additional information from the participants.[53] Data collection ceased at the point of data saturation, the point at which \u201cthe researcher is no longer hearing or seeing new information.\u201d[51][54]\n\nData analysis \nThe study utilized the five steps of the data analysis process from Creswell and Creswell.[55] First, the interview data were organized and prepared for analysis by cataloging interviews by state and type, CPA or CRB owner, and copying or cutting into Excel according to interview question.[51][55] Data was cleansed to eliminate conversation irrelevant to the research study. Interview data was reviewed to allow understanding of the overall ideas from the participants.[55] This immersion into the data helped to ensure a solid understanding prior to detailed analysis.[51]\nInterview transcripts were broken down into similar concepts or codes.[49][55], which were used to identify common themes in the data.[55] Where repetition was identified in the data, themes became apparent.[51] Comparison, or triangulation, between themes identified in the CRB owner interviews and themes identified in the CPA interviews helped to ensure reliability of the research.[53] Due to the small sample size, a Fisher\u2019s exact test calculated using R statistical software was used to determine statistical significance when comparing responses between accountants that serve the industry and those that do not.\n\nResults \nSeveral major themes emerged from the interview data. The primary reason given that CPAs choose to not serve the industry is that cannabis remains illegal at the federal level. All ten of the CRB owners interviewed were able to find a CPA for tax services. The primary theme from CRBs and CPAs who are serving the industry was the need for accountants to have a thorough understanding of the industry and IRS Tax Code Section 280E.\n\nResearch question 1: Why are some CPAs unwilling to service the cannabis industry? \nEight of the CPA respondents indicated that the primary reason to avoid the industry was because the industry is federally illegal and\/or they feared federal criminal prosecution. Four indicated that the special tax requirements and nuances with the tax code were not worth the extra technical training necessary to service what would be a niche industry. In addition, three worried that serving CRBs would jeopardize their CPA license. Table 4 shows the counts and percentages of CPAs responding to the primary reasons they would not serve the cannabis industry.\n\n\n\n\n\n\n\nTable 4. Reasons participating CPAs would not serve the cannabis industry. Note that CPAs may have given more than one reason to not serve the industry; therefore, percentages will not total 100%.\r\n \r\naIncludes COCPA11, who declined an interview but indicated his firm does not serve cannabis clients because it is illegal at the federal level.\n\n\nReason stated\n\nCount (%) of CPA participants who do not serve the industry\n\n\nFederally illegal; fear of criminal prosecution\n\n8 (89%)a\n\n\nDo not want to jeopardize Certified Public Accountant license\n\n3 (33%)\n\n\nComplicated tax requirements; not willing to devote the time and resources to special technical training\n\n4 (44%)\n\n\nBanking\/cash and security issues\n\n1 (11%)\n\n\n\nResearch question 2: How do CRBs compensate for lack of CPA services? \nAll 10 of the CRB owners interviewed indicated that they currently engaged a CPA for their business, though two indicated that they previously had difficulty in finding a CPA who was willing to work with them and were competent. All 10 of the CRBs had CPAs who did their federal income tax filing. Five described their relationship with their CPA as being like an outsourced chief financial officer (CFO).\nFive CPA participants indicated that they do serve the cannabis industry. Three of those five only serve cannabis clients and no other industries. Seven CRB participants and three CPAs who currently serve the industry made a point to note that regardless of the service to be provided, CPAs must have in-depth knowledge of the industry. Tax compliance was the service indicated as being needed most by CRBs from 15 participants. Table 5 summarizes the responses when asked what professional accounting and tax services are most needed in the industry.\n\n\n\n\n\n\n\nTable 5. Professional accounting and tax services perceived as most needed by participating cannabis-related businesses and CPAs. Participants may have identified more than one needed service; therefore, percentages will not total 100%.\n\n\nAccounting and tax services most needed by the cannabis industry\n\nReported by participant type\n\n\nCannabis-related business owners count (%)\n\nCPAs that serve the industry\n\nCPAs that do not serve the industry\n\n\nCPA must know industry in detail regardless of services provided\n\n7 (70%)\n\n3 (60%)\n\n0\n\n\nTax compliance\n\n5 (50%)\n\n5 (100%)\n\n5 (55%)\n\n\nAccounting and tax advice\n\n6 (60%)\n\n3 (60%)\n\n2 (22%)\n\n\nFunction like a CFO\/controller to help management make business decisions\n\n5 (50%)\n\n0\n\n3 (33%)\n\n\nAccounting and bookkeeping\n\n0\n\n3 (60%)\n\n7 (77%)\n\n\nConsulting on areas such as business valuations, M&A, and internal controls\n\n0\n\n1 (20%)\n\n5 (55%)\n\n\nGAAP financial statement audits\n\n0\n\n1 (20%)\n\n3 (33%)\n\n\n\nResearch question 3: What does a CPA need to know prior to engaging to service the industry? \nTable 6 summarizes the CPAs' responses to what is the greatest risk associated with serving the industry. Risk of federal prosecution and loss of CPA license were the two most frequently indicated responses. CPAs who do not serve the industry identified more risks than those who do serve the industry (p = 0.04).\n\n\n\n\n\n\n\nTable 6. Risk in serving the cannabis industry as reported by participating CPAs. Participants may have identified more than one risk factor; therefore, percentages will not total 100%.\n\n\nRisk for CPA serving the industry\n\nReported by participant type\n\n\nCPAs that serve the industry\n\nCPAs that do not serve the industry\n\nTotal CPAs count (%)\n\n\nNothing different from any other client; we are helping them stay in compliance with federal tax law\n\n2 (40%)\n\n0\n\n2 (14%)\n\n\nTax Code Section 280E\/Tax issues\/Tax audits\n\n1 (20%)\n\n1 (11%)\n\n2 (14%)\n\n\nRelying on improper records to support income tax return\n\n1 (20%)\n\n1 (11%)\n\n2 (14%)\n\n\nClient may be doing something outside of regulations\n\n1 (20%)\n\n0\n\n1 (7%)\n\n\nDifficult to recruit staff\n\n1 (20%)\n\n0\n\n1 (7%)\n\n\nGetting sued\/litigation\n\n1 (20%)\n\n1 (11%)\n\n2 (14%)\n\n\nReputational risk\n\n0\n\n2 (22%)\n\n2 (14%)\n\n\nLosing CPA license\n\n0\n\n4 (44%)\n\n4 (29%)\n\n\nFederal prosecution\n\n0\n\n4 (44%)\n\n4 (29%)\n\n\nCash-intensive business\/Security of staff\/Depositing cash receipts from CRBs\n\n0\n\n2 (22%)\n\n2 (14%)\n\n\nLack of technical training on industry\n\n0\n\n2 (22%)\n\n2 (14%)\n\n\n\nAll participants were asked what banking and cash handling issues were unique to the cannabis industry. Ten participants expressed knowledge that most banks will not serve the industry and that those that do charge high fees. Eight participants acknowledged security issues related to having a cash-intensive business. Seven of ten CRBs indicated they had accounts closed at multiple banks and had to find new ones. Four CRBs in remote areas often had to transport cash long distances to a bank that would give them an account. Four CRBs identified alternative ways to bank, including setting up management accounts to handle the cash through a bank, though these often were closed once the bank discovered the transactions were from a cannabis business. Table 7 shows these responses.\n\n\n\n\n\n\n\nTable 7. Banking and cash difficulties for the industry as reported by cannabis-related business owners.\r\n \r\naFirst two digits of participant ID represent the state, next three digits indicate whether the participant was a CRB or CPA, and the last digit represents the number of the interviewee.\n\n\nParticipant IDa\n\nBanking and cash difficulties\n\n\nWACRB1\n\n\n\u2022 Bank is long-distance drive\r\n\n\u2022 Safety\/security issues\r\n\n\u2022 Have had accounts closed\r\n\n\u2022 High fees\r\n\n\u2022 No credit cards\n\n\n\nWACRB2\n\n\n\u2022 Have had 2\u20133 accounts closed\r\n\n\u2022 Significant reporting requirements\/bank audits\n\n\n\nWACRB3\n\n\n\u2022 Bank is long-distance drive\r\n\n\u2022 High fees\r\n\n\u2022 Difficult to keep cash records\r\n\n\u2022 Closed bank account and went back to all cash\n\n\n\nWACRB4\n\n\n\u2022 High fees\r\n\n\u2022 Safety\/security issues\r\n\n\u2022 Have had 3\u20134 accounts closed\r\n\n\u2022 Use multiple accounts\/management company for bank account\n\n\n\nWACRB5\n\n\n\u2022 Have had 3 accounts closed\r\n\n\u2022 Pay as much in cash as possible\r\n\n\u2022 Save up $20s and drive them to pay IRS in cash\r\n\n\u2022 Use management company for bank account\n\n\n\nCOCRB1\n\n\n\u2022 Have had multiple accounts closed\r\n\n\u2022 Bank is long-distance drive\r\n\n\u2022 Hard to pay taxes without bank account\n\n\n\nCOCRB2\n\n\n\u2022 High fees\r\n\n\u2022 Significant reporting requirements\r\n\n\u2022 Bank is long-distance drive\r\n\n\u2022 Safety\/security issues\n\n\n\nCOCRB3\n\n\n\u2022 High fees\r\n\n\u2022 Significant reporting requirements\r\n\n\u2022 Have had multiple accounts closed (even employees\u2019 personal accounts were closed)\n\n\n\nCOCRB4\n\n\n\u2022 Have had account closed\r\n\n\u2022 High fees\r\n\n\u2022 Use multiple accounts\/management company for bank account\n\n\n\nCOCRB5\n\n\n\u2022 High fees\r\n\n\u2022 Pay as much in cash as possible\r\n\n\u2022 Difficult to keep cash records\r\n\n\u2022 Bank \u201cignores\u201d the type of business\/turns a blind eye\n\n\n\n\nThree CPAs that work with the industry and three CRBs also indicated that there was complicated and time-consuming reporting required for cannabis bank accounts. Table 8 reports the responses by CPAs regarding banking and cash issues compared to the responses given by the CRBs. There is not a statistically significant difference between the banking and cash issues identified by accountants that do serve the industry and those that do not (p = 0.35).\n\n\n\n\n\n\n\nTable 8. Banking and cash issues reported by CRBs compared to CPAs. Participants may have identified more than one banking and cash issue; therefore, percentages will not total 100%.\n\n\nBanking and cash issues reported\n\nReported by participant type\n\n\nCRB owner count (%)\n\nCPAs that serve the industry\n\nCPAs that do not serve the industry\n\n\nBanking issues vary by state\n\n0\n\n2 (40%)\n\n0\n\n\nHigh fees\n\n7 (70%)\n\n3 (60%)\n\n0\n\n\nSignificant reporting requirements for bank\n\n3 (30%)\n\n3 (60%)\n\n0\n\n\nSafety\/security issues\n\n3 (30%)\n\n1 (20%)\n\n4 (44%)\n\n\nSome CRBs have bank accounts that are not openly cannabis accounts\n\n4 (40%)\n\n1 (20%)\n\n0\n\n\nNo credit cards\n\n1 (10%)\n\n2 (40%)\n\n0\n\n\nInsider theft\/susceptible to fraud\n\n0\n\n1 (20%)\n\n1 (11%)\n\n\nNo banking available\n\n0\n\n0\n\n4 (44%)\n\n\nComplicated cash record-keeping\n\n2 (20%)\n\n0\n\n2 (22%)\n\n\nBank is long distance drive\n\n4 (40%)\n\n0\n\n0\n\n\nAccounts get closed by bank\n\n7 (70%)\n\n0\n\n0\n\n\n\nAll participants were asked what type of internal controls were needed in the industry to safeguard cash and inventory while ensuring adherence to federal, state, and local regulations. Respondents described physical security controls 18 times. CRB and CPA participants mentioned internal controls for record-keeping requirements needed to support both accounting records and meet state and local regulatory requirements 20 times in the various interviews. There is not a statistically significant difference between internal controls identified by accountants that do serve the industry and those that do not (p = 0.60). Table 9 reports the internal control needs described by interview participants.\n\n\n\n\n\n\n\nTable 9. Internal controls needed to safeguard assets and ensure adherence to regulations in the cannabis industry. Participants may have identified more than one internal control; therefore, percentages will not total 100%.\n\n\nInternal controls needed\n\nReported by participant type\n\n\nCRB owner count (%)\n\nCPAs that serve the industry\n\nCPAs that do not serve the industry\n\n\nPhysical security controls\n\n\n   Cameras\n\n5 (50%)\n\n1 (20%)\n\n2 (22%)\n\n\n   Saves\/Vaults\n\n5 (50%)\n\n0\n\n0\n\n\n   Locks\/Fencing\/Guards\n\n3 (30%)\n\n1 (20%)\n\n1 (11%)\n\n\nSupport record-keeping requirements\n\n\n   Cash logs\/records\/reconciliation\n\n2 (20%)\n\n0\n\n3 (33%)\n\n\n   Frequent inventory counts\/reconciliation\n\n2 (20%)\n\n2 (40%)\n\n3 (33%)\n\n\n   Technology to track inventory and sales\n\n3 (30%)\n\n3 (60%)\n\n2 (22%)\n\n\nSegregation of duties with cash\/inventory handling\n\n3 (30%)\n\n2 (40%)\n\n2 (22%)\n\n\nNo one alone with cash\n\n0\n\n1 (20%)\n\n1 (22%)\n\n\nDrug test employees\n\n0\n\n0\n\n1 (11%)\n\n\nReview\/oversight of classification of costs for Tax Code section 280E\n\n0\n\n0\n\n1 (11%)\n\n\n\nAll CPA participants were asked what special training or technical knowledge is needed to serve the industry. The most common responses indicated that a solid understanding of the industry must include knowing how business operates, what state and local regulations must be followed, and what government and bank reporting requirements exist, as well as knowing detailed knowledge of Tax Code Section 280E. Table 10 reports the special training and technical knowledge CPAs believe is needed to serve the industry competently. While there is not a statistically significant difference between the training and knowledge identified by accountants who do serve the industry and those who do not, it might be significant if there were more data as the p value is just slightly over 0.05 (p = 0.051). Table 11 lists quotes from participants regarding the need for CPAs to have specialized knowledge of the industry.\n\n\n\n\n\n\n\nTable 10. Special training and technical knowledge needed for a CPA to serve the cannabis industry. CPAs may have identified more than one training factor; therefore, percentages will not total 100%.\n\n\nSpecial training and technical knowledge\n\nReported by participant type\n\n\nCPAs that serve the industry\n\nCPAs that do not serve the industry\n\n\nKnowledge of Tax Code Section 280E\n\n4 (80%)\n\n3 (33%)\n\n\nThorough knowledge of cannabis industry\n\n4 (80%)\n\n6 (66%)\n\n\nRead everything available about industry\n\n2 (40%)\n\n0\n\n\nContinuous learning needed\n\n1 (20%)\n\n1 (11%)\n\n\nUnderstanding of cash-intensive business and internal controls\n\n0\n\n1 (11%)\n\n\nUnderstanding of the technology used to track cash and inventory\n\n0\n\n1 (11%)\n\n\nNothing\n\n0\n\n1 (11%)\n\n\n\n\n\n\n\n\n\nTable 11. Participant quotes regarding the need for significant cannabis industry knowledge by CPAs.\r\n \r\naFirst two digits of participant ID represent the state, next three digits indicate whether the participant was a CRB or CPA, and the last digit represents the number of the interviewee.\n\n\nParticipant IDa\n\nQuote\n\n\nCOCPA2\n\n\u201cThere\u2019s a lot of nuance to [the industry] \u2026 where they\u2019re allowed to deduct certain things, not allowed to deduct certain things. \u2026 [A CPA needs] some experience dealing with the companies themselves. They\u2019ve got challenges.\u201d\n\n\nCOCPA5\n\n\u201cThe internal controls, to me, get stepped up to a whole new level when you\u2019re talking about [the cannabis industry]. And so ramping up, at every aspect, whoever\u2019s touching and interfacing with the client, if it\u2019s an audit or most likely tax or advisory \u2026 whoever the team is, making sure they\u2019re well versed in internal controls [for a cannabis business].\u201d\n\n\nCOCPA6\n\n\u201cReally becoming savvy and understanding the industry would be key.\u201d\n\n\nCOCPA7\n\n\u201cYou certainly need to at least be familiar with the business model and industry. \u2026 So really understanding how that business is structured and how it works, and how the industry is structured and how it works is very important.\u201d\n\n\nCOCPA8\n\n\u201cI don't think you can jump into it, not knowing what you're getting into, like even as a bookkeeper. I mean if you got into it, not knowing about 280E, you would drown. You would drown.\u201d\n\n\nWACPA1\n\n\u201cKnowledge of the industry is going to be really, really important.\u201d\n\n\nWACPA2\n\nYou should read that code [280E]. \u2026 Here\u2019s a couple IRS memos \u2026 that you need to read \u2026 here\u2019s some Tax Court cases you need to read \u2026 and they have to understand that like, sort of the sprawling effect of all this 280E stuff is actually a lot more complicated than just cost of goods sold.\u201d\n\n\nWACPA3\n\n\u201cIf you\u2019re going to be in a niche, you really need to understand that business. You need to understand the industry and kind of each of the segments and how they\u2019re interrelated and relate to and interact with each other.\u201d\n\n\nWACRB1\n\n\u201cI think that having a professional CPA that\u2019s very familiar with cannabis law to do your end of the year tax return is very important.\u201d\n\n\nWACRB2\n\n\u201cOh my God, somebody that knows something about the industry. \u2026 You got to know the industry. You got to know about compliance.\u201d\n\n\nWACRB3\n\n\u201c \u2026 having that expertise and that knowledge and that confidence that definitely most businesspeople don\u2019t have, and even [I] \u2026 didn\u2019t have.\u201d\n\n\nCOCRB1\n\n\u201cI would say that if you\u2019re going to work with a traditional bookkeeper, it could be a little challenging because of 280E.\u201d\n\n\nCOCRB3\n\nCRBs need a CPA that \u201cprovides good 280E advice \u2026 know the industry, and 280E and IRS arguments inside and out.\u201d\n\n\nCOCRB4\n\nWe \u201cneed a high-level CPA that knows the industry and has a good connection with an attorney that knows the industry. A CPA needs to be very hands-on in this business to serve as sort of an outsourced CFO to guide operating decisions around 280E.\u201d\n\n\n\nWhen asked to describe their cannabis client acceptance procedures, all five of the CPAs working with the industry indicated that they try to get to know the client much like any other potential engagement. Table 12 summarizes the client acceptance procedures described by CPAs that service the industry.\n\n\n\n\n\n\n\nTable 12. Cannabis client acceptance procedures reportedly in use by CPAs who currently serve the industry. CPAs may have identified more than one procedure; therefore, percentages will not total 100%.\n\n\nClient acceptance procedures in use\n\nCPAs that serve the industry\n\n\nMake clear Tax Code Section 280E expectations\/follow law\/ethics\n\n4 (80%)\n\n\nSpecial language in engagement letter\n\n2 (40%)\n\n\nRequire client to have a bank account\n\n1 (20%)\n\n\nRequire retainer\n\n1 (20%)\n\n\nNot different from any other client\n\n1 (20%)\n\n\n\nDiscussion \nThis study provides a better understanding of the accounting and tax complexities of the cannabis industry as perceived by cannabis business owners and CPAs who are already serving the industry. This study also provides explanation as to why CPAs are reluctant or unwilling to serve the cannabis industry. This study has important implications for the public accounting profession as well as the cannabis industry.\nWhen interviewing a CPA to provide accounting and tax services, there are several characteristics or qualifications that a CRB should consider. The primary consideration for hiring a CPA should be the accountant\u2019s knowledge of the cannabis industry. The CPA needs to have more than just a cursory knowledge of the industry, but rather have an in-depth understanding. For example, the CPA should have a thorough working knowledge of Tax Code Section 280E and internal control issues unique to the industry. The CPA should have attended specific industry training or completed continuing professional education (CPE) courses. The CRB should consider how much CPE the CPA has taken and how recently the CPA attended the CPE. The CRB should consider how many other cannabis clients the accountant already serves or will take on. Finally, the CRB may want to consider how the CPA will be able to assist as the cannabis business matures. CRBs may look to their CPA to provide them financial guidance much like an outsourced CFO.\nA CPA who is thinking about serving the cannabis industry needs to first understand the risks associated with that industry. The CPA participants already serving the cannabis industry did not indicate any particular themes related to risks of serving the industry. In fact, some felt there were no risks that were unique to serving this industry as opposed to others. However, the CPA participants not serving the cannabis industry indicated risks as potential federal prosecution and loss of the CPA license and also identified more risks than those accountants who do serve the industry. It is possible that accountants not serving the cannabis industry may have misconceptions about the industry, influencing the risks they identify, while accountants serving the industry have first-hand knowledge of what the industry is like and where they experience risk versus reward. Last, a CPA should evaluate the need for specialized technical knowledge in tax and accounting for the cannabis industry.\nWhile important with any potential client, client acceptance and continuance procedures must be completed carefully with a potential CRB client. The CPA should start with an interview to get to know the client, and then thoroughly investigate the status of the cannabis license or license application with the state to determine the legality of the business within state and local regulations. The CPA should make the potential client aware of the expectations regarding compliance with all laws and regulations, with specific emphasis on 280E and tax compliance. Additionally, a CPA may wish to learn more about the CRB owner by talking with other business contacts and references. None of the CPA participants in this study conducted background checks for CRB clients, but this could be done if concerns arise. A CPA could also consider collecting a retainer from the CRB client. Finally, the CPA could also modify the engagement letter to clarify expectations regarding staying in compliance with all state and local regulations and federal tax code.\nAny CPA newly entering the cannabis industry should expect continuous learning for the industry. First, they should check with the state board of accountancy to determine if there is any guidance for CPAs in that state related to the cannabis industry. Next, they need to read Code Section 280E in its entirety. The accountant also needs to read all Tax Court cases related to the cannabis industry and understand that the cases thus far have overwhelmingly found against the cannabis businesses. The CPA needs to continually read about new tax cases and other current events related to the industry. There are many CPE courses available to CPAs that are specific to the cannabis industry. The CPA should also consult the AICPA, their state society of CPAs, or look to more mature cannabis states for learning opportunities. For example, the Colorado Society of Certified Public Accountants holds an annual symposium for CPAs interested in the cannabis industry. In addition, the accountant may want to interview a CPA from a more mature state to get advice on how to competently proceed and minimize risk.\nCPAs must also remember that the nuances of accounting for the cannabis industry go beyond Tax Code Section 280E. For example, in newer cannabis states, the CRB may not be able to find a bank account. The CPA needs to be prepared to guide the CRB in establishing effective internal controls that will enable the CRB to safeguard inventory and cash, comply with all state and local regulations, and ensure reliable financial record-keeping of mostly cash transactions. The required internal controls will include, but go far beyond, physical controls such as cameras, vaults, and locks as required by regulations. State regulations also require effective and detailed inventory tracking, usually from seed-to-sale systems. Internal controls will also need to be sufficient to not only safeguard cash and cannabis inventory from theft but also ensure accurate revenue and expense records in sufficient detail for the CPA to properly complete tax returns at the federal, state, and local levels. Once banking is available to the CRB, a new set of reporting requirements will need to be met to maintain the account. Many CRBs will look to their CPA for help with these internal controls and reporting requirements.\nOnly one of the CPA participants from this study provides audit or other attest services to the cannabis industry. In general, there are many CPAs who do not provide any audit or attest services to any industry due to litigation risks. Some states, like Colorado, may at their discretion require a financial statement audit of a CRB by a licensed CPA. In addition, as the cannabis industry evolves, there may be additional funding opportunities that would put cannabis companies on the securities market, as has already taken place in Canada. This would cause these cannabis companies to require annual audited financial statements from a licensed CPA. While this study found there are CPAs that specialize in providing\u2014or are at least willing to provide\u2014accounting and tax services to the cannabis industry, there are not many firms that will do an audit for the industry. This presents additional opportunity for CPAs in the cannabis marketplace.\n\nLimitations \nThis study has several limitations, including limited external validity and potential selection bias. The views of the interviewees may not necessarily generalize to all CRB owners or CPAs in these two states or other states. However, repetition of themes indicating data saturation is a good indication that the responses would be similar if more CRB owners or CBAs were interviewed. Finding participants for the study was difficult as many of those invited declined to participate. The convenience sample drawn could have introduced some level of selection bias into the study. Those who declined participation may have had different views of the industry than those that agreed to participate. In addition, interview responses may be influenced by the interviewer\u2019s presence and some participants may not be \u201cequally articulate and perceptive.\u201d[55] While questions were asked in a neutral tone and data was analyzed without intentional bias, unconscious bias could be present in the study.\n\nRecommendations for future research \nThis study could be expanded to additional states to broaden this field of knowledge. Interviews of participants from states that are newer to the cannabis marketplace may result in findings that are different from the findings of this study of two mature cannabis states. A future study could compare the findings from mature states and developing states. Another interesting follow up study could investigate organization stigma issues for CPA firms working in the cannabis industry.\nQuantitative studies could be designed to obtain data from a larger sample size than is possible with a qualitative study. For example, CPAs could be surveyed to determine their level of agreement with the theory identified herein that the greatest risk in CPAs providing services to CRBs is the risk of federal prosecution. CRB owners could be surveyed to determine their level of agreement with the theory that they need CPAs who have in-depth knowledge of the industry and would be able to perform as an outsourced CFO to guide their business into the future. Another quantitative study could be conducted to further explore the motivational or ethical factors as to why CPAs choose to serve this industry or not. A follow-up quantitative study has already begun in which CPAs have been surveyed to determine their level of agreement with the risks identified in this project and whether they perceive any ethical issues in working with the industry.\n\nConclusion \nThe cannabis industry remains in limbo with state laws in conflict with federal law. The findings of this study indicate that while many CPAs will not serve the cannabis industry, there are competent and knowledgeable CPAs who will. CRB owners need to carefully consider the industry knowledge and experience of a potential CPA prior to engaging them.\nFor CPAs that are considering accepting CRB clients, this study showed that there are risks that should be weighed, such as the risk of federal prosecution and potential loss of the CPA license. Next, the CPA must commit to acquiring and maintaining substantial specialized knowledge related to Tax Code Section 280E, internal controls for a cash-only or cash-intensive business, and the workings of the cannabis industry under the current regulatory conditions. As cannabis legalization continues to expand to new states, the need for CPAs to serve the industry continues to grow. This creates considerable business opportunity for CPAs who are willing to bear the related risk.\n\nAbbreviations, acronyms, and initialisms \nAICPA: American Institute of Certified Public Accountants\nBOA: Board of accountancy\nCBD: Cannabidiol\nCFO: Chief financial officer\nCPA: Certified public accountant\nCRB: Cannabis-related business\nCSA: Controlled Substances Act\nIRB: Institutional review board \nTHC: Tetrahydrocannabinol \n\nSupplementary information \n Appendix A\n Appendix B\nAcknowledgements \nThe primary author acknowledges her doctoral committee from California Southern University for their guidance on this research. The committee included Dr. Stephanie Hoon, Dr. Bruce Gillies, Dr. Mitchell Miller, Dr. Michael Ewald, and Dr. Jennifer Newmann. The author also acknowledges Dr. Richard C. Ott, Associate Professor of Statistics at Colorado Mesa University, for his assistance with the statistical analysis.\n\nAuthor contributions \nThere was one author. The primary author\u2019s doctoral committee from California Southern University is acknowledged. The author(s) read and approved the final manuscript.\n\nEthics approval and consent to participate \nIRB approval was obtained from California Southern University prior to data collection. All participants signed a consent to participate.\n\nFunding \nThe author received no funding for the research.\n\nAvailability of data and materials \nInterview transcripts and manual notes as well as Excel files summarizing data were maintained.\n\nConflict of interest \nThe author has stated no conflict of interest.\n\nReferences \n\n\n\u2191 1.0 1.1 National Conference on State Legislatures (September 2020). \"State medical Marijuana Laws\". https:\/\/www.ncsl.org\/research\/health\/state-medical-marijuana-laws.aspx . Retrieved 22 October 2020 .   \n\n\u2191 Campbell, G. (2012). Pot, Inc.: Inside Medical Marijuana, America's Most Outlaw Industry. Sterling. ISBN 9781402779251.   \n\n\u2191 \"Title 21 United States Code (USC) Controlled Substances Act\". Diversion Control Division. U.S. Department of Justice. https:\/\/www.deadiversion.usdoj.gov\/21cfr\/21usc\/811.htm . Retrieved 20 October 2020 .   \n\n\u2191 \"H.R.1158 - Consolidated Appropriations Act, 2020\". Congress.gov. Library of Congress. 20 December 2019. https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1158\/text . Retrieved 23 October 2020 .   \n\n\u2191 \"H.R.2 - Agriculture Improvement Act of 2018\". Congress.gov. Library of Congress. 20 December 2018. https:\/\/www.congress.gov\/bill\/115th-congress\/house-bill\/2 . Retrieved 23 October 2020 .   \n\n\u2191 Abernethy, A. (25 July 2019). \"Testimony: Hemp Production and the 2018 Farm Bill\". U.S. Food and Drug Administration. https:\/\/www.fda.gov\/news-events\/congressional-testimony\/hemp-production-and-2018-farm-bill-07252019 . Retrieved 05 September 2020 .   \n\n\u2191 7.0 7.1 7.2 Taylor, K.; Bunker, R.B.; Johnson, L.R. et al. (2016). \"An analysis of the accounting and financial effects of inconsistent state and federal laws in the recreational cannabis industry\". Journal of Legal, Ethical and Regulatory Issues 19 (2): 11\u201325. https:\/\/www.abacademies.org\/journals\/month-november-year-2016-vol-19-issue-2-journal-jleri-past-issue.html .   \n\n\u2191 8.0 8.1 8.2 8.3 8.4 8.5 AICPA (14 January 2019). \"An issue brief on state marijuana laws and the CPA profession\" (PDF). AICPA. https:\/\/www.aicpa.org\/advocacy\/state\/downloadabledocuments\/marijuanacpasissuebrief.pdf .   \n\n\u2191 Gottlieb, L.; Munson, M. (2016). \"Insurance Issues Raised by the Legalization of Recreational Marijuana\" (PDF). The Brief 46 (1): 34\u201339. http:\/\/www.bpmlaw.com\/wp-content\/uploads\/2012\/04\/BRF_v046n01_Fall2016_Gottlieb-Munson-2.pdf .   \n\n\u2191 \"26 USC 280E: Expenditures in connection with the illegal sale of drugs\". House.gov. 03 September 1982. https:\/\/uscode.house.gov\/view.xhtml?hl=false&edition=prelim&req=granuleid%3AUSC-prelim-title26-section280E . Retrieved 09 September 2020 .   \n\n\u2191 Alsharaiha, M.N. (2017). \"Smoking Out the Criminals: How Federal Tax Policy Can Reduce Illegal Drug Crime by Supporting the Legal Marijuana Industry\". University of Toledo Law Review 48 (2): 319\u201336. https:\/\/www.utoledo.edu\/law\/studentlife\/lawreview\/volumes\/v48n2.html .   \n\n\u2191 Taillon, B.; Cornelius, J. (2018). \"280E and the Value of Experienced CPAs\". 2018 Marijuana Business Symposium. https:\/\/www.cocpa.org\/product\/2018-marijuana-business-symposium\/ .   \n\n\u2191 \"956. RICO PROSECUTIONS\u201418 U.S.C. \u00a7\u00a7 1961-68\". U.S. Department of Justice Archives. U.S. Department of Justice. 21 January 2020. https:\/\/www.justice.gov\/archives\/jm\/criminal-resource-manual-956-rico-prosecutions-18-usc-1961-68 . Retrieved 23 October 2020 .   \n\n\u2191 Marine, F.J.; Crow, D.E.; Dalton, R.C. et al. (October 2009). \"Criminal RICO: 18 U.S.C. \u00a7\u00a7 1961-1968 - A Manual for Federal Prosecutors, Fifth Revised Edition\" (PDF). U.S. Department of Justice. https:\/\/www.justice.gov\/sites\/default\/files\/usao\/legacy\/2014\/10\/17\/rico.pdf . Retrieved 06 September 2020 .   \n\n\u2191 Reinhart, B.E. (2016). \"Up in Smoke or Down in Flames? A Florida Lawyer's Legal and Ethical Risks in Advising a Marijuana Industry Client\". Florida Bar Journal 90 (3): 20\u201328. https:\/\/www.floridabar.org\/the-florida-bar-journal\/up-in-smoke-or-down-in-flames-a-florida-lawyers-legal-and-ethical-risks-in-advising-a-marijuana-industry-client\/ .   \n\n\u2191 16.0 16.1 Sanders, R.C. (2015). \"To weed or not to weed? The Colorado quandary of legitimate cannabis businesses and the financial institutions who are unable to serve them\". The Penn State Law Review (Summer).   \n\n\u2191 Financial Crimes Enforcement Network. \"What we do\". FINCEN.   \n\n\u2191 Houser, K.A.; Rosacker, R.E. (2015). \"High times: A history of cannabis laws in the United States\". International Journal of Business and Public Administration 12 (2). http:\/\/www.iabpad.com\/high-times-a-history-of-marijuana-laws-in-the-united-states\/ .   \n\n\u2191 Shu-Acquaye, F. (2016). \"The Unintended Consequence to Legalizing Marijuana Use: The Banking Conondrum\". Cleveland State Law Review 64 (2): 315. https:\/\/engagedscholarship.csuohio.edu\/clevstlrev\/vol64\/iss2\/10 .   \n\n\u2191 Dayton, M.R. (2017). \"The great unknown: Trump\u2019s impact on cannabis banking\". Marijuana Venture (3): 142\u20136. https:\/\/www.marijuanaventure.com\/product\/mv-march-2017\/ .   \n\n\u2191 Financial Crimes Enforcement Network (30 June 2020). \"Marijuana Banking Update\" (PDF). https:\/\/www.fincen.gov\/sites\/default\/files\/shared\/508_295174_MJ%20Banking%20Update%203rd%20QTR%20FY2020_Public_508%20compliant.pdf . Retrieved 05 September 2020 .   \n\n\u2191 \"H.R.1588 - Ending Federal Marijuana Prohibition Act of 2019\". Congress.gov. Library of Congress. 08 April 2019. https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1588 . Retrieved 23 October 2020 .   \n\n\u2191 \"H.R.1595 - Secure And Fair Enforcement Banking Act of 2019\". Congress.gov. Library of Congress. 26 September 2019. https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1595 . Retrieved 23 October 2020 .   \n\n\u2191 \"H.R.2703 - MAPLE Act of 2019\". Congress.gov. Library of Congress. 26 June 2019. https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/2703 . Retrieved 23 October 2020 .   \n\n\u2191 \"H.R.3884 - MORE Act of 2020\". Congress.gov. Library of Congress. 07 December 2020. https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/3884 . Retrieved 23 October 2020 .   \n\n\u2191 Madison, C.J. (01 August 2018). \"Inventory accounting for cannabis businesses: Sec. 280E and the impact of tax reform\". The Tax Advisor. https:\/\/www.thetaxadviser.com\/issues\/2018\/aug\/inventory-accounting-cannabis-businesses.html .   \n\n\u2191 27.0 27.1 Department of Revenue, Marijuana Enforcement Division (2016). \"Medical Marijuana Rules - 1 CCR 212 - Version ID 6821\" (PDF). State of Colorado. https:\/\/www.sos.state.co.us\/CCR\/GenerateRulePdf.do?ruleVersionId=6821 . Retrieved 23 October 2020 .   \n\n\u2191 28.0 28.1 28.2 28.3 Department of Revenue, Marijuana Enforcement Division (2017). \"Medical Marijuana Rules - 1 CCR 212 - Version ID 7094\" (PDF). State of Colorado. https:\/\/www.sos.state.co.us\/CCR\/GenerateRulePdf.do?ruleVersionId=7094 . Retrieved 23 October 2020 .   \n\n\u2191 \"Chapter 314-55 WAC, Marijuana Licenses, Application Process, Requirements, and Reporting\". Washington State Legislature. State of Washington. 2020. https:\/\/apps.leg.wa.gov\/wac\/default.aspx?cite=314-55 .   \n\n\u2191 30.0 30.1 30.2 Frazer, L. (2016). \"Internal Control: Is it a Benefit or Fad to Small Companies? A Literature Dependency Perspective\". Journal of Accounting and Finance 16 (4): 149\u201361. https:\/\/articlegateway.com\/index.php\/JAF\/article\/view\/1047 .   \n\n\u2191 Khairul, M.Z.; Anuar, N.; Ahmad, S.A.P.S. (2016). \"Internal controls and fraud \u2013 empirical evidence from oil and gas company\". Journal of Financial Crime 23 (4): 1154\u201368. doi:10.1108\/JFC-04-2016-0021.   \n\n\u2191 32.0 32.1 Hunzicker, A. (2018). \"Solving financial mistakes made by cannabis businesses\". Marijuana Venture (8): 148\u201350. https:\/\/www.marijuanaventure.com\/product\/mv-august-2018\/ .   \n\n\u2191 Lee, D. (01 August 2018). \"Big opportunities in cannabis clients, compliance\". Accounting Today. https:\/\/www.accountingtoday.com\/news\/big-opportunities-in-cannabis-clients-compliance?tag=00000157-6c2f-dfbe-a35f-fcaf52df0000 . Retrieved 05 September 2020 .   \n\n\u2191 34.0 34.1 Kapp, L.A.; Heslop, G. (2011). \"Protecting Small Businesses from Fraud\". The CPA Journal 81 (10): 62\u20137. https:\/\/www.nysscpa.org\/news\/publications\/the-cpa-journal\/article-preview?ArticleID=10702 .   \n\n\u2191 \"Cash Intensive Businesses Audit Techniques Guide - Table of Contents\". Internal Revenue Service. April 2010. https:\/\/www.irs.gov\/businesses\/small-businesses-self-employed\/cash-intensive-businesses-audit-techniques-guide-table-of-contents . Retrieved 05 September 2020 .   \n\n\u2191 \"Cash Intensive Businesses Audit Techniques - Chapter 1: Introduction and Overview of the Cash Intensive Business\" (PDF). Internal Revenue Service. April 2010. https:\/\/www.irs.gov\/pub\/irs-utl\/cashchapter1_210627.pdf . Retrieved 05 September 2020 .   \n\n\u2191 37.0 37.1 Satnaliwala, M. (2018). \"Don't Overlook Physical Access: Internal auditors need to evaluate how facilities are protected as part of the organization's overall security efforts\". Internal Auditor 75 (5): 22\u20133. https:\/\/go.gale.com\/ps\/anonymous?id=GALE .   \n\n\u2191 Arens, A.A.; Elder, R.J.; Beasley, M.S. et al. (2017). Auditing and Assurance Services: An Integrated Approach (16th ed.). Pearson. ISBN 9780134065823.   \n\n\u2191 Simkins, M.G.; Norman, C.A.S.; Rose, J.G. (2014). Core Concepts of Accounting Information Systems (13th ed.). Wiley. ISBN 9781119033288.   \n\n\u2191 AICPA (15 December 2014). \"Code of Professional Conduct\". AICPA. https:\/\/pub.aicpa.org\/codeofconduct\/Ethics.aspx . Retrieved 05 September 2020 .   \n\n\u2191 41.0 41.1 Satterlund, C. (15 March 2018). \"CPAs and Cannabis\". Washington State Board of Accountancy. https:\/\/acb.wa.gov\/cpas-and-cannabis . Retrieved 05 September 2020 .   \n\n\u2191 42.0 42.1 42.2 42.3 42.4 42.5 AICPA (January 2019). \"Providing services to businesses in the marijuana industry: A sample of current board positions\" (PDF). AICPA. https:\/\/www.aicpa.org\/advocacy\/state\/downloadabledocuments\/marijuana-state-board-positions.pdf . Retrieved 22 October 2020 .   \n\n\u2191 43.0 43.1 43.2 Sterna, S.; Wolfe, J. (2017). \"Liability Risks and Other Concerns when Servicing Marijuana Businesses\". The CPA Journal 87 (10): 9\u201310. https:\/\/www.cpajournal.com\/2017\/10\/19\/liability-risks-concerns-servicing-marijuana-businesses\/ .   \n\n\u2191 Ohanesian, M.V. (01 March 2018). \"Preparing to work indirectly with the cannabis industry\". The Tax Adviser. https:\/\/www.thetaxadviser.com\/issues\/2018\/mar\/preparing-work-indirectly-cannabis-industry.html .   \n\n\u2191 45.0 45.1 Devers, C.E.; Dewett, T.; Mishina, Y. et al. (2009). \"A General Theory of Organizational Stigma\". Organization Science 20 (1): 154\u2013171. doi:10.1287\/orsc.1080.0367.   \n\n\u2191 46.0 46.1 Hudson, B.A.; Okhuysen, G.A. (2009). \"Not with a Ten-Foot Pole: Core Stigma, Stigma Transfer, and Improbable Persistence of Men's Bathhouses\". Organization Science 20 (1): 134\u201353. doi:10.1287\/orsc.1080.0368.   \n\n\u2191 Hampel, C.E.; Tracey, P. (2016). \"How Organizations Move from Stigma to Legitimacy: The Case of Cook\u2019s Travel Agency in Victorian Britain\". Academy of Management Journal 60 (6): 2175\u2013207. doi:10.5465\/amj.2015.0365.   \n\n\u2191 Gaetano, C. (2017). \"NYSSCPA Marijuana Symposium addresses risks and benefits of the industry\" (PDF). The Trusted Professional 20 (1): 1. https:\/\/www.nysscpa.org\/docs\/default-source\/trusted-professional-archives\/tp_jan_feb_17.pdf .   \n\n\u2191 49.0 49.1 Belotto, M.J. (2018). \"Data Analysis Methods for Qualitative Research: Managing the Challenges of Coding, Interrater Reliability, and Thematic Analysis\". The Qualitative Report 23 (11): 2622-2633. https:\/\/nsuworks.nova.edu\/tqr\/vol23\/iss11\/2 .   \n\n\u2191 50.0 50.1 Astalin, P.K. (2013). \"Qualitative Research Designs: A Conceptual Framework\". International Journal of Social Sciences & Interdisciplinary Research 2 (1): 118\u201324. http:\/\/www.i-scholar.in\/index.php\/ijssir\/article\/view\/43645 .   \n\n\u2191 51.0 51.1 51.2 51.3 51.4 51.5 Savin-Baden, M.; Major, C.H. (2013). Qualitative Research: The Essential Guide to Theory and Practice. Routledge. ISBN 9780415674782.   \n\n\u2191 Cooper, D.R.; Schindler, P.S. (2013). Business Research Methods (12th ed.). McGraw-Hill Education. ISBN 9780073521503.   \n\n\u2191 53.0 53.1 Merriam, S.B. (2009). Qualitative Research: A Guide to Design and Implementation (3rd ed.). Jossey-Bass. ISBN 9780470283547.   \n\n\u2191 Astroth, K.S. (2018). \"Focusing on the Fundamentals: Reading Qualitative Research with a Critical Eye\". Nephrology Nursing Journal 45 (4): 381\u201386. PMID 30303648.   \n\n\u2191 55.0 55.1 55.2 55.3 55.4 55.5 Creswell, J.W.; Creswell, J.D. (2018). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (5th ed.). SAGE Publications, Inc. ISBN 9781506386706.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; this version lists them in order of appearance, by design. Two of the original references for AICPA CPA docs had broken URLs; what is presumed to be the same documents\u2014although updated for 2019\u2014were found elsewhere on the AICPA site and used for this version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\">https:\/\/www.limswiki.org\/index.php\/Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on cannabis banking and financesLIMSwiki journal articles on laws and regulations\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 February 2021, at 17:47.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 73 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","a40f37542af50527bcd32bf00da69900_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Accounting_and_the_US_cannabis_industry_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Accounting and the US cannabis industry: Federal financial regulations and the perspectives of certified public accountants and cannabis businesses owners<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis\" data-key=\"ae3a33525e4682427d4498e16c586f9e\">Cannabis<\/a>-related businesses (CRBs), in states where cannabis is legal, may be unable to obtain professional financial services such as banking, insurance, and accounting because of federal laws and regulations. This qualitative study investigated the following research questions.\n<\/p>\n<dl><dd>1. Why are some certified public accountants (CPAs) unwilling to provide services to cannabis-related businesses?<\/dd>\n<dd>2. How do CRBs compensate for lack of CPA services?<\/dd>\n<dd>3. What does a CPA need to know about the cannabis industry prior to engaging to provide services to CRBs?<\/dd><\/dl>\n<p><b>Methods<\/b>: Data for this grounded-theory qualitative study was gathered from twenty-three semi-structured phone and face-to-face interviews. Ten cannabis-related business owners were recruited from a convenience sample after attempting a broad recruiting effort. Thirteen CPAs with active licenses in Colorado or Washington State participated from firms of varying size and willingness to serve the cannabis industry. The individual interviews, which lasted from twenty minutes to more than an hour, focused on the participants\u2019 perceptions of the complexities of accounting and tax compliance for cannabis businesses.\n<\/p><p><b>Results<\/b>: Eight of the thirteen CPAs interviewed would not provide services to the cannabis industry with the primary reason given that cannabis is federally illegal. All ten of the cannabis business owners interviewed indicated they engage a CPA to provide tax services. Seven out of ten CRB participants and ten of the thirteen CPA participants indicated that extensive industry knowledge is needed for an accountant to competently provide services to a CRB.\n<\/p><p><b>Conclusions<\/b>: CRB owners need to carefully consider the industry knowledge and experience of a potential CPA prior to engaging them. This study shows that U.S. CPAs should weigh the risk of federal prosecution and potential loss of the CPA license when deciding whether to serve a CRB client. The study also found that a CPA must commit to acquiring and maintaining substantial specialized knowledge related to tax Code Section 280E, internal controls for a cash-only or cash-intensive business, and the workings of the cannabis industry under the current regulatory conditions.\n<\/p><p><b>Keywords<\/b>: cannabis, marijuana, accounting, business, certified public accountants, financial regulation, Colorado, Washington State\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>There are a growing number of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis\" data-key=\"ae3a33525e4682427d4498e16c586f9e\">cannabis<\/a>-related businesses (CRBs) in the United States, as many states have legalized cannabis for medical and\/or recreational use even though it remains illegal at the federal level under the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Controlled_Substances_Act\" class=\"extiw wiki-link\" title=\"wikipedia:Controlled Substances Act\" data-key=\"dc9b3ee3a1db6f2642725d442821d995\">Controlled Substances Act<\/a> (CSA). Thirty-three states plus the District of Columbia, Puerto Rico, the U.S. Virgin Islands, and Guam have legalized <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_(drug)\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis (drug)\" data-key=\"43fbd34351979f1e17186f202a2b1e49\">medical cannabis<\/a>, while eleven states plus the District of Columbia and the Northern Mariana Islands have also legalized recreational cannabis usage as of September 2020.<sup id=\"rdp-ebb-cite_ref-NCSLState20_1-0\" class=\"reference\"><a href=\"#cite_note-NCSLState20-1\">[1]<\/a><\/sup> Table 1 shows the legalized states by category. Cannabis has been listed as a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Controlled_Substances_Act#Schedules_of_controlled_substances\" class=\"extiw wiki-link\" title=\"wikipedia:Controlled Substances Act\" data-key=\"377b47481b93c475f21024bb54934989\">Schedule I<\/a> drug since 1970.<sup id=\"rdp-ebb-cite_ref-CampbellPot12_2-0\" class=\"reference\"><a href=\"#cite_note-CampbellPot12-2\">[2]<\/a><\/sup> Regardless of state laws, cannabis is a Schedule I drug according to the federal government, and those found trafficking in cannabis could face criminal prosecution.<sup id=\"rdp-ebb-cite_ref-DOJControlled_3-0\" class=\"reference\"><a href=\"#cite_note-DOJControlled-3\">[3]<\/a><\/sup> The Rohrabacher-Blumenauer amendment prohibits the use of federal funds to prosecute a cannabis-related business activity that operates within \u201cStates that have legalized the use of medical marijuana,\u201d and was most recently extended to the 2019\u20132020 fiscal year in the 2020 Consolidated Appropriations Act.<sup id=\"rdp-ebb-cite_ref-CongressHR1158_4-0\" class=\"reference\"><a href=\"#cite_note-CongressHR1158-4\">[4]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 1.<\/b> States in which cannabis is legal for medical and\/or recreational use. Abbreviations are standard US Postal Service abbreviations.<sup id=\"rdp-ebb-cite_ref-NCSLState20_1-1\" class=\"reference\"><a href=\"#cite_note-NCSLState20-1\">[1]<\/a><\/sup> CBD = cannabidiol; THC =tetrahydrocannabinol.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">States in which cannabis is legal only for medical use\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">States in which cannabis is legal for medical and recreational use\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">States in which only CBD (low THC) cannabis is allowed\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">States in which cannabis remains illegal\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AR, AZ, CT, DC, DE, FL, HI, LA, MD, MN, MO, MT, ND, NH, NJ, NM, NY, OH, OK, PA, RI, UT, and WV\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AK, CA, CO, IL, MA, ME, MI, NV, OR, VT, and WA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AL, GA, IA, IN, KY, MS, NC, SC, TN, TX, VA, WI, and WY\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ID, KS, NE, SD\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Cannabis and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hemp\" class=\"extiw wiki-link\" title=\"wikipedia:Hemp\" data-key=\"2e896be8a228178ae105f8b468061a0f\">hemp<\/a> are both varieties of the cannabis plant but with different <a href=\"https:\/\/en.wikipedia.org\/wiki\/Tetrahydrocannabinol\" class=\"extiw wiki-link\" title=\"wikipedia:Tetrahydrocannabinol\" data-key=\"c63b7f849adf168f4b4ff293132f1e53\">tetrahydrocannabinol<\/a> (THC) content. Hemp products became federally legal with the passage of the [[wikipedia:[Agriculture Improvement Act of 2018|Agriculture Improvement Act of 2018]].<sup id=\"rdp-ebb-cite_ref-CongressHR2_5-0\" class=\"reference\"><a href=\"#cite_note-CongressHR2-5\">[5]<\/a><\/sup> The bill \u201cremoved hemp, defined as cannabis (<i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_sativa\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis sativa\" data-key=\"618f17d4f6938557aacc017ee0f52bbd\">Cannabis sativa<\/a><\/i> L.) and derivatives of cannabis with extremely low concentrations of the psychoactive compound delta-9-tetrahydrocannabinol (THC) (no more than 0.3 percent THC on a dry weight basis), from the definition of marijuana in the Controlled Substances Act (CSA).\u201d<sup id=\"rdp-ebb-cite_ref-USFDAHempProd19_6-0\" class=\"reference\"><a href=\"#cite_note-USFDAHempProd19-6\">[6]<\/a><\/sup> Cannabis remains federally illegal and is the topic of this study.\n<\/p><p>This conflict between federal and state laws is problematic because cannabis-related businesses and ancillary businesses operating in legalized states and complying with all state laws are unable to fully comply with federal laws. Many cannabis businesses in states where cannabis is legal are facing difficulties in obtaining professional financial services because cannabis is still a controlled substance at the federal level.<sup id=\"rdp-ebb-cite_ref-TaylorAnAnal16_7-0\" class=\"reference\"><a href=\"#cite_note-TaylorAnAnal16-7\">[7]<\/a><\/sup> For example, CRBs face difficulty obtaining banking services such as checking, credit cards, electronic transfers, and loans, which results in cash-only or cash-intensive business operation.<sup id=\"rdp-ebb-cite_ref-TaylorAnAnal16_7-1\" class=\"reference\"><a href=\"#cite_note-TaylorAnAnal16-7\">[7]<\/a><\/sup> As such, many certified public accountants (CPAs) may be unwilling to provide their accounting and tax services to CRBs due to increased risks associated with the industry.<sup id=\"rdp-ebb-cite_ref-TaylorAnAnal16_7-2\" class=\"reference\"><a href=\"#cite_note-TaylorAnAnal16-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AICPAAnIssue19_8-0\" class=\"reference\"><a href=\"#cite_note-AICPAAnIssue19-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GottliebInsurance16_9-0\" class=\"reference\"><a href=\"#cite_note-GottliebInsurance16-9\">[9]<\/a><\/sup>\n<\/p><p>Because cannabis is a Schedule I drug, a cannabis business is subject to Internal Revenue Service (IRS) Code Section 280E, which disallows the deduction of ordinary business expenses in arriving at taxable income.<sup id=\"rdp-ebb-cite_ref-HouseUSC280E_10-0\" class=\"reference\"><a href=\"#cite_note-HouseUSC280E-10\">[10]<\/a><\/sup> As such, Code Section 280E results in significantly higher effective tax rates for cannabis businesses than for other businesses.<sup id=\"rdp-ebb-cite_ref-AlsharaihaSmoking17_11-0\" class=\"reference\"><a href=\"#cite_note-AlsharaihaSmoking17-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Taillon280E18_12-0\" class=\"reference\"><a href=\"#cite_note-Taillon280E18-12\">[12]<\/a><\/sup> The Racketeer Influenced and Corrupt Organizations Act (RICO) of 1970 provides for federal criminal prosecution of individuals involved with criminal activities, including drug trafficking.<sup id=\"rdp-ebb-cite_ref-DoJRICO_13-0\" class=\"reference\"><a href=\"#cite_note-DoJRICO-13\">[13]<\/a><\/sup> One does not have to be directly involved with the criminal enterprise to be prosecuted under RICO, as it applies to others associated with the criminal enterprise, include those providing professional services such as accountants, lawyers, and bankers.<sup id=\"rdp-ebb-cite_ref-DOJCriminalRICOPDF09_14-0\" class=\"reference\"><a href=\"#cite_note-DOJCriminalRICOPDF09-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ReinhartUpIn16_15-0\" class=\"reference\"><a href=\"#cite_note-ReinhartUpIn16-15\">[15]<\/a><\/sup>\n<\/p><p>Cannabis businesses are also greatly affected by the Bank Secrecy Act (BSA) and anti-money laundering regulations that are designed to help identify and report money laundering activity.<sup id=\"rdp-ebb-cite_ref-SandersToWeed15_16-0\" class=\"reference\"><a href=\"#cite_note-SandersToWeed15-16\">[16]<\/a><\/sup> The Financial Crimes Enforcement Network (FinCEN), a bureau of the U.S. Department of the Treasury, exists to \u201csafeguard the financial system from illicit use and combat money laundering.\u201d<sup id=\"rdp-ebb-cite_ref-FCENAbout_17-0\" class=\"reference\"><a href=\"#cite_note-FCENAbout-17\">[17]<\/a><\/sup> In practical terms, FinCEN guidance stated that providing banking services to cannabis businesses was illegal and required banks to self-report this federally illegal activity.<sup id=\"rdp-ebb-cite_ref-HouserHigh15_18-0\" class=\"reference\"><a href=\"#cite_note-HouserHigh15-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Shu-AcquayeTheUnint16_19-0\" class=\"reference\"><a href=\"#cite_note-Shu-AcquayeTheUnint16-19\">[19]<\/a><\/sup> Due to the BSA risks and FinCEN reporting requirements, most large banks will not provide accounts for cannabis-related businesses.<sup id=\"rdp-ebb-cite_ref-DraytonTheGreat17_20-0\" class=\"reference\"><a href=\"#cite_note-DraytonTheGreat17-20\">[20]<\/a><\/sup> However, there is evidence that some banks and credit unions are serving the industry. According to FinCEN, as of June 2020, 695 financial institutions in the U.S. are reportedly serving the cannabis industry; this is down from a high of 747 institutions in November 2019.<sup id=\"rdp-ebb-cite_ref-FINCENMarij20_21-0\" class=\"reference\"><a href=\"#cite_note-FINCENMarij20-21\">[21]<\/a><\/sup>\n<\/p><p>Bills have been proposed that would change the federal landscape for the cannabis industry. For example, H.R. 1588, the Ending Federal Marijuana Prohibition Act of 2019<sup id=\"rdp-ebb-cite_ref-CongressHR1588_22-0\" class=\"reference\"><a href=\"#cite_note-CongressHR1588-22\">[22]<\/a><\/sup>, was introduced in the House of Representatives in March of 2019 and referred to the House Energy and Commerce and House Judiciary committees. There have been several attempts to pass legislation to ease the banking burden for the industry, including the Secure and Fair Enforcement (SAFE) Banking Act of 2019, which passed in the House of Representatives but has not yet passed the Senate.<sup id=\"rdp-ebb-cite_ref-CongressHR1595_23-0\" class=\"reference\"><a href=\"#cite_note-CongressHR1595-23\">[23]<\/a><\/sup> There have been numerous cannabis-related bills introduced in the 116th Congress in 2019\u20132020, ranging from the MAPLE Act, which would remove cannabis from the list of crimes that would prevent an immigrant entrance into the U.S., to the MORE Act, which provides for the total decriminalization of cannabis and expungement of cannabis-related convictions. However, as of October 2020, none of these bills have passed both chambers.<sup id=\"rdp-ebb-cite_ref-CongressHR2703_24-0\" class=\"reference\"><a href=\"#cite_note-CongressHR2703-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CongressHR3884_25-0\" class=\"reference\"><a href=\"#cite_note-CongressHR3884-25\">[25]<\/a><\/sup> Changes in federal legislation could potentially reduce, if not eliminate, the business problem studied herein. For example, if federal prohibition were eliminated, cannabis clients would be no different than any other client in a legal industry from the perspective of a CPA. Passage of a cannabis banking act would allow the industry to utilize regular banking services and eliminate some risk associated with a cash-intensive type of business.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Accounting_and_internal_control\">Accounting and internal control<\/span><\/h3>\n<p>There are several unique accounting issues that affect the cannabis industry. Due to the state and local regulatory requirements, CRBs need to maintain effective accounting records. Tax Code Section 280E requires attention to inventory accounting while following the tax requirements.<sup id=\"rdp-ebb-cite_ref-MadisonInvent18_26-0\" class=\"reference\"><a href=\"#cite_note-MadisonInvent18-26\">[26]<\/a><\/sup> Lack of banking availability requires special attention to internal controls to safeguard cash and maintain good accounting records of cash transactions. State regulations also require thorough accounting records as well as strong internal controls.<sup id=\"rdp-ebb-cite_ref-CODeptRevMedical16_27-0\" class=\"reference\"><a href=\"#cite_note-CODeptRevMedical16-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CODeptRevMedical17_28-0\" class=\"reference\"><a href=\"#cite_note-CODeptRevMedical17-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WSL314-55_29-0\" class=\"reference\"><a href=\"#cite_note-WSL314-55-29\">[29]<\/a><\/sup>\n<\/p><p>Internal controls are also important to a cannabis business. These are the activities that an organization takes to provide reasonable assurance that the company can meet its goals.<sup id=\"rdp-ebb-cite_ref-FrazerInternal16_30-0\" class=\"reference\"><a href=\"#cite_note-FrazerInternal16-30\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KhairulInternal16_31-0\" class=\"reference\"><a href=\"#cite_note-KhairulInternal16-31\">[31]<\/a><\/sup> Failure to comply with state, local, or tax regulations can result in penalties, fines, and potential revocation of the cannabis license.<sup id=\"rdp-ebb-cite_ref-HunzickerSolv18_32-0\" class=\"reference\"><a href=\"#cite_note-HunzickerSolv18-32\">[32]<\/a><\/sup> For example, effective internal controls were missing in the Alterman v. Commissioner case in which a medical cannabis company failed to maintain adequate records and failed to understand how to comply with Tax Code Section 280E.<sup id=\"rdp-ebb-cite_ref-LeeBigOpp18_33-0\" class=\"reference\"><a href=\"#cite_note-LeeBigOpp18-33\">[33]<\/a><\/sup> Due to complexities in accounting and regulation, cannabis businesses should be perpetually ready for a potential audit by regulators or the IRS.<sup id=\"rdp-ebb-cite_ref-HunzickerSolv18_32-1\" class=\"reference\"><a href=\"#cite_note-HunzickerSolv18-32\">[32]<\/a><\/sup> CPAs can provide small businesses, including cannabis businesses, with guidance to develop and implement effective internal controls.<sup id=\"rdp-ebb-cite_ref-KappProtect11_34-0\" class=\"reference\"><a href=\"#cite_note-KappProtect11-34\">[34]<\/a><\/sup>\n<\/p><p>A cash-intensive business requires some unique internal controls to safeguard the asset and to maintain adequate records of cash transactions.<sup id=\"rdp-ebb-cite_ref-FrazerInternal16_30-1\" class=\"reference\"><a href=\"#cite_note-FrazerInternal16-30\">[30]<\/a><\/sup> The IRS defines a cash-intensive business as one that receives most of its revenues in cash and\/or pays many of its expenses in cash.<sup id=\"rdp-ebb-cite_ref-IRSCash10_35-0\" class=\"reference\"><a href=\"#cite_note-IRSCash10-35\">[35]<\/a><\/sup> Cash may be stolen by employees and is most susceptible to theft when entering or exiting the business.<sup id=\"rdp-ebb-cite_ref-FrazerInternal16_30-2\" class=\"reference\"><a href=\"#cite_note-FrazerInternal16-30\">[30]<\/a><\/sup> To maintain proper accounting records to support potential tax and other regulatory audits, a cash-intensive business should \u201cdocument the flow of each receipt or revenue from the customer\u2019s hands to the business, to the final end in the business bank account or as payment for a business expense.\u201d<sup id=\"rdp-ebb-cite_ref-IRSCash10Chpt1_36-0\" class=\"reference\"><a href=\"#cite_note-IRSCash10Chpt1-36\">[36]<\/a><\/sup>\n<\/p><p>Physical controls may be one of the most crucial internal controls, and security must be considered for areas \u201cto include property, office buildings, warehouses, utility rooms \u2026 and vehicles as well as employees, contractors, and visitors.\u201d<sup id=\"rdp-ebb-cite_ref-SatnaliwalaDont18_37-0\" class=\"reference\"><a href=\"#cite_note-SatnaliwalaDont18-37\">[37]<\/a><\/sup> Physical controls over cash and cannabis inventory include keeping them in a safe or vault, video surveillance, and promptly depositing cash into a bank account if the cannabis business has one.<sup id=\"rdp-ebb-cite_ref-CODeptRevMedical17_28-1\" class=\"reference\"><a href=\"#cite_note-CODeptRevMedical17-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArensAudit17_38-0\" class=\"reference\"><a href=\"#cite_note-ArensAudit17-38\">[38]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SimkinCore14_39-0\" class=\"reference\"><a href=\"#cite_note-SimkinCore14-39\">[39]<\/a><\/sup> Physical controls should be tested frequently to ensure continuous safeguarding of assets.<sup id=\"rdp-ebb-cite_ref-SatnaliwalaDont18_37-1\" class=\"reference\"><a href=\"#cite_note-SatnaliwalaDont18-37\">[37]<\/a><\/sup> Frequent physical inventory counts are needed, and differences between the actual count and inventory records should be examined right away.<sup id=\"rdp-ebb-cite_ref-KappProtect11_34-1\" class=\"reference\"><a href=\"#cite_note-KappProtect11-34\">[34]<\/a><\/sup> Colorado, for examples, has regulations that require daily reconciliation of inventory on-hand to the inventory tracking system.<sup id=\"rdp-ebb-cite_ref-CODeptRevMedical16_27-1\" class=\"reference\"><a href=\"#cite_note-CODeptRevMedical16-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CODeptRevMedical17_28-2\" class=\"reference\"><a href=\"#cite_note-CODeptRevMedical17-28\">[28]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"CPA_requirements_and_risks\">CPA requirements and risks<\/span><\/h3>\n<p>CPAs must adhere to an extremely high standard of ethical conduct as provided by the American Institute of Certified Public Accountants (AICPA) Code of Professional Conduct, as well as any individual state codes of conduct.<sup id=\"rdp-ebb-cite_ref-AICPACode14_40-0\" class=\"reference\"><a href=\"#cite_note-AICPACode14-40\">[40]<\/a><\/sup> Unlike many business professionals, CPAs obtain a license or permit to practice from the state(s) in which they provide services.<sup id=\"rdp-ebb-cite_ref-AICPAAnIssue19_8-1\" class=\"reference\"><a href=\"#cite_note-AICPAAnIssue19-8\">[8]<\/a><\/sup> A violation of ethics or \u201clack of good moral character\u201d could potentially result in a loss of the CPA license, and therefore the potential loss of the ability to earn a living.<sup id=\"rdp-ebb-cite_ref-AICPAAnIssue19_8-2\" class=\"reference\"><a href=\"#cite_note-AICPAAnIssue19-8\">[8]<\/a><\/sup>\n<\/p><p>Most state boards of accountancy have provided somewhat unclear official guidance on the provision of services for the cannabis industry.<sup id=\"rdp-ebb-cite_ref-AICPAAnIssue19_8-3\" class=\"reference\"><a href=\"#cite_note-AICPAAnIssue19-8\">[8]<\/a><\/sup> The Washington State Board of Accountancy (BOA) stated in 2014, and again in 2018, that a CPA\u2019s provision of services to a cannabis-related business does not constitute a violation of the BOA\u2019s rules. This statement followed the March 2018 signing of the Engrossed Substitute Senate Bill 5928, which states that the provision of services to a CRB by a CPA does not, by itself, constitute a crime.<sup id=\"rdp-ebb-cite_ref-SatterlundCPA18_41-0\" class=\"reference\"><a href=\"#cite_note-SatterlundCPA18-41\">[41]<\/a><\/sup> The Washington BOA further recommended that CPAs consider the risks associated with serving the cannabis industry and that CPAs engage an attorney for counsel.<sup id=\"rdp-ebb-cite_ref-SatterlundCPA18_41-1\" class=\"reference\"><a href=\"#cite_note-SatterlundCPA18-41\">[41]<\/a><\/sup> As for Colorado, their BOA issued a position statement on December 16, 2015 that indicated the provision of services by a Certified Public Accountant to a cannabis business is not \u201cspecifically prohibited by the Accountancy Act.\u201d<sup id=\"rdp-ebb-cite_ref-AICPAProviding19_42-0\" class=\"reference\"><a href=\"#cite_note-AICPAProviding19-42\">[42]<\/a><\/sup> The Colorado Board went on to caution CPAs that their position statement does not constitute an endorsement for CPAs to enter the industry.<sup id=\"rdp-ebb-cite_ref-AICPAProviding19_42-1\" class=\"reference\"><a href=\"#cite_note-AICPAProviding19-42\">[42]<\/a><\/sup>\n<\/p><p>In January of 2019, the National Association of State Boards of Accountancy (NASBA), in conjunction with the AICPA, published a document entitled \"Providing services to businesses in the marijuana industry: A sample of current board positions.\"<sup id=\"rdp-ebb-cite_ref-AICPAProviding19_42-2\" class=\"reference\"><a href=\"#cite_note-AICPAProviding19-42\">[42]<\/a><\/sup> This paper summarized the board positions for Alaska, Arizona, Arkansas, Colorado, Connecticut, Florida, Iowa, Maryland, Massachusetts, Michigan, New Mexico, Nevada, Oregon, and Washington. This sample includes states in which cannabis is legal only for medical purposes, states in which cannabis is legal for recreational or medical usage, and a state in which cannabis remains prohibited. All state boards in the report, except New Mexico, indicated that the CPA would not face disciplinary action by the board for providing services to a cannabis business, assuming the CPA was in compliance with all state laws.<sup id=\"rdp-ebb-cite_ref-AICPAProviding19_42-3\" class=\"reference\"><a href=\"#cite_note-AICPAProviding19-42\">[42]<\/a><\/sup> This was even the case for Iowa, where only low-THC, high-CBD (<a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabidiol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabidiol\" data-key=\"0c46162c3d7b370d7448646c25334265\">cannabidiol<\/a>) cannabis is allowed; the Iowa BOA indicated that Iowa CPAs serving cannabis businesses in legalized states would not face disciplinary action.<sup id=\"rdp-ebb-cite_ref-AICPAProviding19_42-4\" class=\"reference\"><a href=\"#cite_note-AICPAProviding19-42\">[42]<\/a><\/sup>\n<\/p><p>While providing services to cannabis businesses does not by itself necessarily constitute a violation of good moral character, other problems can result from a CPA serving the cannabis industry.<sup id=\"rdp-ebb-cite_ref-SternaLiab17_43-0\" class=\"reference\"><a href=\"#cite_note-SternaLiab17-43\">[43]<\/a><\/sup> In some cases, a CPA may be found to have \u201caided and abetted\u201d or been involved in a \u201cconspiracy to violate\u201d the federal CSA or racketeering laws.<sup id=\"rdp-ebb-cite_ref-SternaLiab17_43-1\" class=\"reference\"><a href=\"#cite_note-SternaLiab17-43\">[43]<\/a><\/sup> A CPA may also be exposed to criminal investigation and\/or prosecution as well as potential fines, penalties, and sanctions if he participates in \u201cdishonest, fraudulent, or criminal acts\u201d associated with the cannabis industry.<sup id=\"rdp-ebb-cite_ref-SternaLiab17_43-2\" class=\"reference\"><a href=\"#cite_note-SternaLiab17-43\">[43]<\/a><\/sup>\n<\/p><p>Cannabis businesses have a higher likelihood of income tax audit than other businesses due to complexities of Tax Code Section 280E.<sup id=\"rdp-ebb-cite_ref-OhanesianPrep18_44-0\" class=\"reference\"><a href=\"#cite_note-OhanesianPrep18-44\">[44]<\/a><\/sup> The large number of cash transactions may result in the absence of a clear paper trail, which increases the risk of tax evasion and makes cannabis businesses targets for federal tax audits.<sup id=\"rdp-ebb-cite_ref-SandersToWeed15_16-1\" class=\"reference\"><a href=\"#cite_note-SandersToWeed15-16\">[16]<\/a><\/sup> Some state cannabis regulations\u2014including those found in New Mexico, Minnesota, and Colorado\u2014may require a financial statement audit of a cannabis business to be completed by a CPA.<sup id=\"rdp-ebb-cite_ref-AICPAAnIssue19_8-4\" class=\"reference\"><a href=\"#cite_note-AICPAAnIssue19-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CODeptRevMedical17_28-3\" class=\"reference\"><a href=\"#cite_note-CODeptRevMedical17-28\">[28]<\/a><\/sup>\n<\/p><p>Additionally, CPAs may believe associating with the cannabis industry could damage their reputation in the business community with current or prospective clients. Devers <i>et al.<\/i> have shown that reputation of a firm can indeed be seen by outsiders as an indication of the firm\u2019s quality of services.<sup id=\"rdp-ebb-cite_ref-DeversAGen08_45-0\" class=\"reference\"><a href=\"#cite_note-DeversAGen08-45\">[45]<\/a><\/sup> CPA firms may also be viewed as less than legitimate based on their overall association with the somewhat controversial cannabis industry.<sup id=\"rdp-ebb-cite_ref-DeversAGen08_45-1\" class=\"reference\"><a href=\"#cite_note-DeversAGen08-45\">[45]<\/a><\/sup> Core-stigmatized organizations are those for whom outsiders have a \u201cperceived violation of social norms\u201d and may be looked at unfavorably.<sup id=\"rdp-ebb-cite_ref-HudsonNotWith09_46-0\" class=\"reference\"><a href=\"#cite_note-HudsonNotWith09-46\">[46]<\/a><\/sup> Current or prospective clients may avoid associating with a CPA firm who works in the cannabis industry because they worry that negative stigma may transfer to them.<sup id=\"rdp-ebb-cite_ref-HampelHow16_47-0\" class=\"reference\"><a href=\"#cite_note-HampelHow16-47\">[47]<\/a><\/sup> However, some CPAs may determine that they are willing to accept such core stigma as part of their business strategy, as may be the case for CPA firms who specialize in cannabis clients.<sup id=\"rdp-ebb-cite_ref-HudsonNotWith09_46-1\" class=\"reference\"><a href=\"#cite_note-HudsonNotWith09-46\">[46]<\/a><\/sup>\n<\/p><p>Lastly, professional standards require that a CPA only take on engagements for which he or she has the appropriate technical knowledge about the industry to complete the work competently.<sup id=\"rdp-ebb-cite_ref-AICPAProviding19_42-5\" class=\"reference\"><a href=\"#cite_note-AICPAProviding19-42\">[42]<\/a><\/sup> The complexities of the cannabis industry make meeting this standard costly and not necessarily worth the risks to many CPAs.<sup id=\"rdp-ebb-cite_ref-AICPAAnIssue19_8-5\" class=\"reference\"><a href=\"#cite_note-AICPAAnIssue19-8\">[8]<\/a><\/sup> In general, CPAs need to evaluate the many risks associated with providing services to the industry and may determine that the risks outweigh the benefits and choose to not take on cannabis clients.<sup id=\"rdp-ebb-cite_ref-GaetanoNYSSCPA17_48-0\" class=\"reference\"><a href=\"#cite_note-GaetanoNYSSCPA17-48\">[48]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Purpose_of_this_study\">Purpose of this study<\/span><\/h3>\n<p>Given the previously discussed, this qualitative study investigated the following research questions:\n<\/p>\n<dl><dd>1. Why are some certified public accountants (CPAs) unwilling to provide services to cannabis-related businesses?<\/dd>\n<dd>2. How do CRBs compensate for lack of CPA services?<\/dd>\n<dd>3. What does a CPA need to know about the cannabis industry prior to engaging to provide services to CRBs?<\/dd><\/dl>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>Qualitative research is particularly well-suited for the \u201cearly stages of research\u201d as in the case of a young industry such as cannabis.<sup id=\"rdp-ebb-cite_ref-BelottoData18_49-0\" class=\"reference\"><a href=\"#cite_note-BelottoData18-49\">[49]<\/a><\/sup> The grounded theory approach to qualitative research utilizes \u201csystematical methodological procedures\u201d to identify theories as they \u201cemerge from the data.\u201d<sup id=\"rdp-ebb-cite_ref-AstalinQual13_50-0\" class=\"reference\"><a href=\"#cite_note-AstalinQual13-50\">[50]<\/a><\/sup> Grounded theory is particularly appropriate for research such as this study, which seeks to generate theory based upon the reported data from participants.<sup id=\"rdp-ebb-cite_ref-Savin-BadenQual13_51-0\" class=\"reference\"><a href=\"#cite_note-Savin-BadenQual13-51\">[51]<\/a><\/sup> Grounded theory requires one to continuously compare and analyze data previously collected to the new data being collected to identify patterns and themes, and produce theories. Interviews of participants are often used as a method of data collection in grounded theory research.<sup id=\"rdp-ebb-cite_ref-AstalinQual13_50-1\" class=\"reference\"><a href=\"#cite_note-AstalinQual13-50\">[50]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Savin-BadenQual13_51-1\" class=\"reference\"><a href=\"#cite_note-Savin-BadenQual13-51\">[51]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CooperBusi13_52-0\" class=\"reference\"><a href=\"#cite_note-CooperBusi13-52\">[52]<\/a><\/sup>\n<\/p><p>Semi-structured interviews of cannabis business owners were used to learn about the financial difficulties CRBs face, particularly related to accounting and finding CPA services. Semi-structured interviews of CPAs provided an understanding of the willingness or hesitancy to provide services to CRBs, the potential risks related to the industry, and accounting and tax issues relevant to CRBs. The interview questions for CPAs may be found in the supplementary information as Additional file 1: Appendix A, and the questions for CRB owners as Additional file 2: Appendix B. Institutional review board (IRB) approval was obtained from California Southern University prior to data collection. Signed statements of informed consent from all participants were emailed or hand delivered during face-to-face interviews. Interviews were one-on-one. Nineteen interviews were conducted via telephone and four conducted in-person between February 2019 and May 2019. Interviews lasted from approximately 20 minutes to more than an hour.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Participants\">Participants<\/span><\/h3>\n<p>A list of 3,076 active CPA firms was downloaded from the Colorado and Washington State Board of Accountancy websites on December 2 and 3, 2018, respectively. The listed firms\u2019 websites were then reviewed to locate contact information for partners and managers, who were then emailed a recruitment letter. This was a highly manual process and not all websites had contact information for their employees posted, resulting in less than half the number of email contacts compared to the number of firms. While 1,249 requests for participation were emailed, only four willing participants responded. A convenience sample was recruited for the remaining CPA participants by posting requests for participation on LinkedIn and Facebook and through conversations and emails with accounting colleagues in both states. Deliberate effort was made to include participants from a variety of sizes of firms as well as from firms who did serve the cannabis industry and firms that did not. No participants withdrew during the interviews; however, one CPA declined the interview and simply provided a statement that his firm does not provide services to the cannabis industry because it remains federally illegal.\n<\/p><p>Colorado and Washington State both maintain publicly available listings of licensed cannabis businesses. The Colorado Department of Revenue publishes lists of licensed cannabis facilities on their state website. The listings reported approximately 2,200 licensed Colorado retail stores, retail manufacturers, retail cultivators, and medical cultivators for December 2018. These lists contained only company name, city, zip code, and license number. Seventy randomly selected companies from the combined lists were researched on the internet to locate email or phone information. A request for participation was emailed if an email address was listed or if there was a \u201ccontact us\u201d option on the website.\n<\/p><p>The Washington State Liquor and Cannabis Board publishes a report of monthly \u201cSales Activity by License Number.\u201d For December 2018, 433 cannabis licenses reported sales activity. The 433 license numbers were then cross referenced to the \u201cWashington Listing of Marijuana Applicants\u201d report which provides company name, address, and phone number. Since phone numbers were readily available on this report, phone calls were made to request participation from randomly selected licensees throughout the list.\n<\/p><p>Forty-five licensed cannabis businesses were called and seventy were emailed to request owner participation, but finding CRB participants proved to be extremely difficult. No emails were answered. Phone calls never got past the employee answering the phone, meaning the employee indicated they would not be interested or would leave a message for the owner to respond. Only two CRB owners responded, but they both declined participation and generally expressed unease in sharing any information related to their business with a stranger. It became apparent that obtaining participation in this manner was not effective.\n<\/p><p>Next, requests for participation were posted on LinkedIn and Facebook and requested through conversations with colleagues to identify and recruit potential CRB owners in both states. This led to five leads where there was a personal relationship between someone in our professional network and a CRB owner. One of those five, a small Colorado retailer, did not agree to participate as they were uncomfortable sharing their business information. The other four CRB owners agreed to participate and provided additional names of CRB owners in their networks to contact at the conclusion of their interviews. This process of referrals continued until the study reached ten CRB participants. No participants withdrew during the interviews. The unresponsive CRBs varied in terms of location and type of firm, as did those CRBs that participated. Tables 2 and 3 show the descriptive data for the participants.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 2.<\/b> Overview of CRBs participating in interviews.<br \/> <br \/><sup>a<\/sup>First two digits of participant ID represent the state, next three digits indicate the participant was a CRB owner, and last digit represents the number of the interviewee.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant ID<sup>a<\/sup>\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Type of cannabis business\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Year established\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Legal structure\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Approximate 2018 revenues\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Producer\/Processor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2014\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$3.5M\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Producer\/Processor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2015\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$600k\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Producer\/Processor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2015\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partnership\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$5k (closing)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Processor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2013\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Declined\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Retailer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2017\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partnership\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$5.6M\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Retailer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2014\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$2.5M\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Retailer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2014\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$4.0M\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Producer\/Processor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2014\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">S-Corp\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$5.0 to 6.0M\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Producer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2018\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">S-Corp\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$0 (start-up)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Retailer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2009 (medical)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LLC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">$1.0M\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 3.<\/b> Overview of CPAs participating in interviews.<br \/> <br \/><sup>a<\/sup>First two digits of participant ID represent the state, second three digits indicate the participant was a Certified Public Accountant, and last digit represents the number of the interviewee.<br \/><sup>b<\/sup>Choices were: single CPA; local CPA firm with multiple CPAs; regional CPA firm; national CPA firm; Big 4 CPA firm; or not in public accounting.<br \/><sup>c<\/sup>COCPA11 was a partner at a Big 4 accounting firm that declined to do a phone interview but responded via email that the firm\u2019s policy is to not serve cannabis businesses because they are illegal at the federal level.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant ID<sup>a<\/sup>\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Partner or manager within firm\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Primary county or US Zip code of practice\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Type of firm<sup>b<\/sup>\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Do you or your firm provide services to CRBs?\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACPA1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98501\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Local\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACPA2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Various\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Single CPA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACPA3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98004\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Local\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81501\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Single CPA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80237\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">National\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nationwide\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">National\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Manager\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81501\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">National\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81501\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Regional\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80433\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Local\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80433 and 81501\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Local\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80901\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Single CPA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Denver\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Regional\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Manager\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81521\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Local\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Partner\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nationwide\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Big 4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Data_collection\">Data collection<\/span><\/h3>\n<p>Three central research questions were the focus of the study, with sub-questions asked to narrow the focus as the interview was conducted. As interviews took place, responses were analyzed for clarity and understanding. At times, probes were asked to \u201cfollow up something already asked\u201d and to seek additional information from the participants.<sup id=\"rdp-ebb-cite_ref-MerriamQual09_53-0\" class=\"reference\"><a href=\"#cite_note-MerriamQual09-53\">[53]<\/a><\/sup> Data collection ceased at the point of data saturation, the point at which \u201cthe researcher is no longer hearing or seeing new information.\u201d<sup id=\"rdp-ebb-cite_ref-Savin-BadenQual13_51-2\" class=\"reference\"><a href=\"#cite_note-Savin-BadenQual13-51\">[51]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AstrothFocus18_54-0\" class=\"reference\"><a href=\"#cite_note-AstrothFocus18-54\">[54]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_analysis\">Data analysis<\/span><\/h3>\n<p>The study utilized the five steps of the data analysis process from Creswell and Creswell.<sup id=\"rdp-ebb-cite_ref-CreswellResearch18_55-0\" class=\"reference\"><a href=\"#cite_note-CreswellResearch18-55\">[55]<\/a><\/sup> First, the interview data were organized and prepared for analysis by cataloging interviews by state and type, CPA or CRB owner, and copying or cutting into Excel according to interview question.<sup id=\"rdp-ebb-cite_ref-Savin-BadenQual13_51-3\" class=\"reference\"><a href=\"#cite_note-Savin-BadenQual13-51\">[51]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CreswellResearch18_55-1\" class=\"reference\"><a href=\"#cite_note-CreswellResearch18-55\">[55]<\/a><\/sup> Data was cleansed to eliminate conversation irrelevant to the research study. Interview data was reviewed to allow understanding of the overall ideas from the participants.<sup id=\"rdp-ebb-cite_ref-CreswellResearch18_55-2\" class=\"reference\"><a href=\"#cite_note-CreswellResearch18-55\">[55]<\/a><\/sup> This immersion into the data helped to ensure a solid understanding prior to detailed analysis.<sup id=\"rdp-ebb-cite_ref-Savin-BadenQual13_51-4\" class=\"reference\"><a href=\"#cite_note-Savin-BadenQual13-51\">[51]<\/a><\/sup>\n<\/p><p>Interview transcripts were broken down into similar concepts or codes.<sup id=\"rdp-ebb-cite_ref-BelottoData18_49-1\" class=\"reference\"><a href=\"#cite_note-BelottoData18-49\">[49]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CreswellResearch18_55-3\" class=\"reference\"><a href=\"#cite_note-CreswellResearch18-55\">[55]<\/a><\/sup>, which were used to identify common themes in the data.<sup id=\"rdp-ebb-cite_ref-CreswellResearch18_55-4\" class=\"reference\"><a href=\"#cite_note-CreswellResearch18-55\">[55]<\/a><\/sup> Where repetition was identified in the data, themes became apparent.<sup id=\"rdp-ebb-cite_ref-Savin-BadenQual13_51-5\" class=\"reference\"><a href=\"#cite_note-Savin-BadenQual13-51\">[51]<\/a><\/sup> Comparison, or triangulation, between themes identified in the CRB owner interviews and themes identified in the CPA interviews helped to ensure reliability of the research.<sup id=\"rdp-ebb-cite_ref-MerriamQual09_53-1\" class=\"reference\"><a href=\"#cite_note-MerriamQual09-53\">[53]<\/a><\/sup> Due to the small sample size, a Fisher\u2019s exact test calculated using R statistical software was used to determine statistical significance when comparing responses between accountants that serve the industry and those that do not.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>Several major themes emerged from the interview data. The primary reason given that CPAs choose to not serve the industry is that cannabis remains illegal at the federal level. All ten of the CRB owners interviewed were able to find a CPA for tax services. The primary theme from CRBs and CPAs who are serving the industry was the need for accountants to have a thorough understanding of the industry and IRS Tax Code Section 280E.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Research_question_1:_Why_are_some_CPAs_unwilling_to_service_the_cannabis_industry.3F\">Research question 1: Why are some CPAs unwilling to service the cannabis industry?<\/span><\/h3>\n<p>Eight of the CPA respondents indicated that the primary reason to avoid the industry was because the industry is federally illegal and\/or they feared federal criminal prosecution. Four indicated that the special tax requirements and nuances with the tax code were not worth the extra technical training necessary to service what would be a niche industry. In addition, three worried that serving CRBs would jeopardize their CPA license. Table 4 shows the counts and percentages of CPAs responding to the primary reasons they would not serve the cannabis industry.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 4.<\/b> Reasons participating CPAs would not serve the cannabis industry. Note that CPAs may have given more than one reason to not serve the industry; therefore, percentages will not total 100%.<br \/> <br \/><sup>a<\/sup>Includes COCPA11, who declined an interview but indicated his firm does not serve cannabis clients because it is illegal at the federal level.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Reason stated\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Count (%) of CPA participants who do not serve the industry\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Federally illegal; fear of criminal prosecution\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8 (89%)<sup>a<\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Do not want to jeopardize Certified Public Accountant license\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (33%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Complicated tax requirements; not willing to devote the time and resources to special technical training\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (44%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Banking\/cash and security issues\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Research_question_2:_How_do_CRBs_compensate_for_lack_of_CPA_services.3F\">Research question 2: How do CRBs compensate for lack of CPA services?<\/span><\/h3>\n<p>All 10 of the CRB owners interviewed indicated that they currently engaged a CPA for their business, though two indicated that they previously had difficulty in finding a CPA who was willing to work with them and were competent. All 10 of the CRBs had CPAs who did their federal income tax filing. Five described their relationship with their CPA as being like an outsourced chief financial officer (CFO).\n<\/p><p>Five CPA participants indicated that they do serve the cannabis industry. Three of those five only serve cannabis clients and no other industries. Seven CRB participants and three CPAs who currently serve the industry made a point to note that regardless of the service to be provided, CPAs must have in-depth knowledge of the industry. Tax compliance was the service indicated as being needed most by CRBs from 15 participants. Table 5 summarizes the responses when asked what professional accounting and tax services are most needed in the industry.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 5.<\/b> Professional accounting and tax services perceived as most needed by participating cannabis-related businesses and CPAs. Participants may have identified more than one needed service; therefore, percentages will not total 100%.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Accounting and tax services most needed by the cannabis industry\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"4\">Reported by participant type\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Cannabis-related business owners count (%)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that serve the industry\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that do not serve the industry\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CPA must know industry in detail regardless of services provided\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7 (70%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (60%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tax compliance\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 (50%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 (100%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 (55%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Accounting and tax advice\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6 (60%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (60%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (22%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Function like a CFO\/controller to help management make business decisions\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 (50%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (33%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Accounting and bookkeeping\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (60%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7 (77%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Consulting on areas such as business valuations, M&A, and internal controls\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 (55%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GAAP financial statement audits\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (33%)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Research_question_3:_What_does_a_CPA_need_to_know_prior_to_engaging_to_service_the_industry.3F\">Research question 3: What does a CPA need to know prior to engaging to service the industry?<\/span><\/h3>\n<p>Table 6 summarizes the CPAs' responses to what is the greatest risk associated with serving the industry. Risk of federal prosecution and loss of CPA license were the two most frequently indicated responses. CPAs who do not serve the industry identified more risks than those who do serve the industry (<i>p<\/i> = 0.04).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 6.<\/b> Risk in serving the cannabis industry as reported by participating CPAs. Participants may have identified more than one risk factor; therefore, percentages will not total 100%.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Risk for CPA serving the industry\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"4\">Reported by participant type\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that serve the industry\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that do not serve the industry\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Total CPAs count (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nothing different from any other client; we are helping them stay in compliance with federal tax law\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (40%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (14%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tax Code Section 280E\/Tax issues\/Tax audits\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (14%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Relying on improper records to support income tax return\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (14%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Client may be doing something outside of regulations\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (7%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Difficult to recruit staff\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (7%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Getting sued\/litigation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (14%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reputational risk\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (22%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (14%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Losing CPA license\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (44%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (29%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Federal prosecution\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (44%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (29%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cash-intensive business\/Security of staff\/Depositing cash receipts from CRBs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (22%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (14%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lack of technical training on industry\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (22%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (14%)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>All participants were asked what banking and cash handling issues were unique to the cannabis industry. Ten participants expressed knowledge that most banks will not serve the industry and that those that do charge high fees. Eight participants acknowledged security issues related to having a cash-intensive business. Seven of ten CRBs indicated they had accounts closed at multiple banks and had to find new ones. Four CRBs in remote areas often had to transport cash long distances to a bank that would give them an account. Four CRBs identified alternative ways to bank, including setting up management accounts to handle the cash through a bank, though these often were closed once the bank discovered the transactions were from a cannabis business. Table 7 shows these responses.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 7.<\/b> Banking and cash difficulties for the industry as reported by cannabis-related business owners.<br \/> <br \/><sup>a<\/sup>First two digits of participant ID represent the state, next three digits indicate whether the participant was a CRB or CPA, and the last digit represents the number of the interviewee.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant ID<sup>a<\/sup>\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Banking and cash difficulties\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 Bank is long-distance drive<br \/>\n\u2022 Safety\/security issues<br \/>\n\u2022 Have had accounts closed<br \/>\n\u2022 High fees<br \/>\n\u2022 No credit cards\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 Have had 2\u20133 accounts closed<br \/>\n\u2022 Significant reporting requirements\/bank audits\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 Bank is long-distance drive<br \/>\n\u2022 High fees<br \/>\n\u2022 Difficult to keep cash records<br \/>\n\u2022 Closed bank account and went back to all cash\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 High fees<br \/>\n\u2022 Safety\/security issues<br \/>\n\u2022 Have had 3\u20134 accounts closed<br \/>\n\u2022 Use multiple accounts\/management company for bank account\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 Have had 3 accounts closed<br \/>\n\u2022 Pay as much in cash as possible<br \/>\n\u2022 Save up $20s and drive them to pay IRS in cash<br \/>\n\u2022 Use management company for bank account\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 Have had multiple accounts closed<br \/>\n\u2022 Bank is long-distance drive<br \/>\n\u2022 Hard to pay taxes without bank account\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 High fees<br \/>\n\u2022 Significant reporting requirements<br \/>\n\u2022 Bank is long-distance drive<br \/>\n\u2022 Safety\/security issues\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 High fees<br \/>\n\u2022 Significant reporting requirements<br \/>\n\u2022 Have had multiple accounts closed (even employees\u2019 personal accounts were closed)\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 Have had account closed<br \/>\n\u2022 High fees<br \/>\n\u2022 Use multiple accounts\/management company for bank account\n<\/p>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<p>\u2022 High fees<br \/>\n\u2022 Pay as much in cash as possible<br \/>\n\u2022 Difficult to keep cash records<br \/>\n\u2022 Bank \u201cignores\u201d the type of business\/turns a blind eye\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Three CPAs that work with the industry and three CRBs also indicated that there was complicated and time-consuming reporting required for cannabis bank accounts. Table 8 reports the responses by CPAs regarding banking and cash issues compared to the responses given by the CRBs. There is not a statistically significant difference between the banking and cash issues identified by accountants that do serve the industry and those that do not (<i>p<\/i> = 0.35).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 8.<\/b> Banking and cash issues reported by CRBs compared to CPAs. Participants may have identified more than one banking and cash issue; therefore, percentages will not total 100%.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Banking and cash issues reported\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"4\">Reported by participant type\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CRB owner count (%)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that serve the industry\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that do not serve the industry\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Banking issues vary by state\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (40%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High fees\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7 (70%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (60%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Significant reporting requirements for bank\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (30%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (60%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Safety\/security issues\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (30%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (44%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Some CRBs have bank accounts that are not openly cannabis accounts\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (40%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No credit cards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (10%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (40%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Insider theft\/susceptible to fraud\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No banking available\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (44%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Complicated cash record-keeping\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (22%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bank is long distance drive\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (40%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Accounts get closed by bank\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7 (70%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>All participants were asked what type of internal controls were needed in the industry to safeguard cash and inventory while ensuring adherence to federal, state, and local regulations. Respondents described physical security controls 18 times. CRB and CPA participants mentioned internal controls for record-keeping requirements needed to support both accounting records and meet state and local regulatory requirements 20 times in the various interviews. There is not a statistically significant difference between internal controls identified by accountants that do serve the industry and those that do not (<i>p<\/i> = 0.60). Table 9 reports the internal control needs described by interview participants.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 9.<\/b> Internal controls needed to safeguard assets and ensure adherence to regulations in the cannabis industry. Participants may have identified more than one internal control; therefore, percentages will not total 100%.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Internal controls needed\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"4\">Reported by participant type\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CRB owner count (%)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that serve the industry\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that do not serve the industry\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\">Physical security controls\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">   Cameras\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 (50%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (22%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">   Saves\/Vaults\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 (50%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">   Locks\/Fencing\/Guards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (30%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\">Support record-keeping requirements\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">   Cash logs\/records\/reconciliation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (33%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">   Frequent inventory counts\/reconciliation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (40%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (33%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">   Technology to track inventory and sales\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (30%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (60%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (22%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Segregation of duties with cash\/inventory handling\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (30%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (40%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (22%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No one alone with cash\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (22%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Drug test employees\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Review\/oversight of classification of costs for Tax Code section 280E\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>All CPA participants were asked what special training or technical knowledge is needed to serve the industry. The most common responses indicated that a solid understanding of the industry must include knowing how business operates, what state and local regulations must be followed, and what government and bank reporting requirements exist, as well as knowing detailed knowledge of Tax Code Section 280E. Table 10 reports the special training and technical knowledge CPAs believe is needed to serve the industry competently. While there is not a statistically significant difference between the training and knowledge identified by accountants who do serve the industry and those who do not, it might be significant if there were more data as the <i>p<\/i> value is just slightly over 0.05 (<i>p<\/i> = 0.051). Table 11 lists quotes from participants regarding the need for CPAs to have specialized knowledge of the industry.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 10.<\/b> Special training and technical knowledge needed for a CPA to serve the cannabis industry. CPAs may have identified more than one training factor; therefore, percentages will not total 100%.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Special training and technical knowledge\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"3\">Reported by participant type\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that serve the industry\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that do not serve the industry\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Knowledge of Tax Code Section 280E\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (80%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (33%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Thorough knowledge of cannabis industry\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (80%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6 (66%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Read everything available about industry\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (40%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Continuous learning needed\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Understanding of cash-intensive business and internal controls\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Understanding of the technology used to track cash and inventory\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nothing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (11%)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 11.<\/b> Participant quotes regarding the need for significant cannabis industry knowledge by CPAs.<br \/> <br \/><sup>a<\/sup>First two digits of participant ID represent the state, next three digits indicate whether the participant was a CRB or CPA, and the last digit represents the number of the interviewee.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Participant ID<sup>a<\/sup>\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Quote\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cThere\u2019s a lot of nuance to [the industry] \u2026 where they\u2019re allowed to deduct certain things, not allowed to deduct certain things. \u2026 [A CPA needs] some experience dealing with the companies themselves. They\u2019ve got challenges.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cThe internal controls, to me, get stepped up to a whole new level when you\u2019re talking about [the cannabis industry]. And so ramping up, at every aspect, whoever\u2019s touching and interfacing with the client, if it\u2019s an audit or most likely tax or advisory \u2026 whoever the team is, making sure they\u2019re well versed in internal controls [for a cannabis business].\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cReally becoming savvy and understanding the industry would be key.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cYou certainly need to at least be familiar with the business model and industry. \u2026 So really understanding how that business is structured and how it works, and how the industry is structured and how it works is very important.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCPA8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cI don't think you can jump into it, not knowing what you're getting into, like even as a bookkeeper. I mean if you got into it, not knowing about 280E, you would drown. You would drown.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACPA1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cKnowledge of the industry is going to be really, really important.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACPA2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">You should read that code [280E]. \u2026 Here\u2019s a couple IRS memos \u2026 that you need to read \u2026 here\u2019s some Tax Court cases you need to read \u2026 and they have to understand that like, sort of the sprawling effect of all this 280E stuff is actually a lot more complicated than just cost of goods sold.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACPA3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cIf you\u2019re going to be in a niche, you really need to understand that business. You need to understand the industry and kind of each of the segments and how they\u2019re interrelated and relate to and interact with each other.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cI think that having a professional CPA that\u2019s very familiar with cannabis law to do your end of the year tax return is very important.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cOh my God, somebody that knows something about the industry. \u2026 You got to know the industry. You got to know about compliance.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WACRB3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201c \u2026 having that expertise and that knowledge and that confidence that definitely most businesspeople don\u2019t have, and even [I] \u2026 didn\u2019t have.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cI would say that if you\u2019re going to work with a traditional bookkeeper, it could be a little challenging because of 280E.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CRBs need a CPA that \u201cprovides good 280E advice \u2026 know the industry, and 280E and IRS arguments inside and out.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COCRB4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">We \u201cneed a high-level CPA that knows the industry and has a good connection with an attorney that knows the industry. A CPA needs to be very hands-on in this business to serve as sort of an outsourced CFO to guide operating decisions around 280E.\u201d\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>When asked to describe their cannabis client acceptance procedures, all five of the CPAs working with the industry indicated that they try to get to know the client much like any other potential engagement. Table 12 summarizes the client acceptance procedures described by CPAs that service the industry.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 12.<\/b> Cannabis client acceptance procedures reportedly in use by CPAs who currently serve the industry. CPAs may have identified more than one procedure; therefore, percentages will not total 100%.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Client acceptance procedures in use\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CPAs that serve the industry\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Make clear Tax Code Section 280E expectations\/follow law\/ethics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (80%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Special language in engagement letter\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (40%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Require client to have a bank account\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Require retainer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not different from any other client\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (20%)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>This study provides a better understanding of the accounting and tax complexities of the cannabis industry as perceived by cannabis business owners and CPAs who are already serving the industry. This study also provides explanation as to why CPAs are reluctant or unwilling to serve the cannabis industry. This study has important implications for the public accounting profession as well as the cannabis industry.\n<\/p><p>When interviewing a CPA to provide accounting and tax services, there are several characteristics or qualifications that a CRB should consider. The primary consideration for hiring a CPA should be the accountant\u2019s knowledge of the cannabis industry. The CPA needs to have more than just a cursory knowledge of the industry, but rather have an in-depth understanding. For example, the CPA should have a thorough working knowledge of Tax Code Section 280E and internal control issues unique to the industry. The CPA should have attended specific industry training or completed continuing professional education (CPE) courses. The CRB should consider how much CPE the CPA has taken and how recently the CPA attended the CPE. The CRB should consider how many other cannabis clients the accountant already serves or will take on. Finally, the CRB may want to consider how the CPA will be able to assist as the cannabis business matures. CRBs may look to their CPA to provide them financial guidance much like an outsourced CFO.\n<\/p><p>A CPA who is thinking about serving the cannabis industry needs to first understand the risks associated with that industry. The CPA participants already serving the cannabis industry did not indicate any particular themes related to risks of serving the industry. In fact, some felt there were no risks that were unique to serving this industry as opposed to others. However, the CPA participants not serving the cannabis industry indicated risks as potential federal prosecution and loss of the CPA license and also identified more risks than those accountants who do serve the industry. It is possible that accountants not serving the cannabis industry may have misconceptions about the industry, influencing the risks they identify, while accountants serving the industry have first-hand knowledge of what the industry is like and where they experience risk versus reward. Last, a CPA should evaluate the need for specialized technical knowledge in tax and accounting for the cannabis industry.\n<\/p><p>While important with any potential client, client acceptance and continuance procedures must be completed carefully with a potential CRB client. The CPA should start with an interview to get to know the client, and then thoroughly investigate the status of the cannabis license or license application with the state to determine the legality of the business within state and local regulations. The CPA should make the potential client aware of the expectations regarding compliance with all laws and regulations, with specific emphasis on 280E and tax compliance. Additionally, a CPA may wish to learn more about the CRB owner by talking with other business contacts and references. None of the CPA participants in this study conducted background checks for CRB clients, but this could be done if concerns arise. A CPA could also consider collecting a retainer from the CRB client. Finally, the CPA could also modify the engagement letter to clarify expectations regarding staying in compliance with all state and local regulations and federal tax code.\n<\/p><p>Any CPA newly entering the cannabis industry should expect continuous learning for the industry. First, they should check with the state board of accountancy to determine if there is any guidance for CPAs in that state related to the cannabis industry. Next, they need to read Code Section 280E in its entirety. The accountant also needs to read all Tax Court cases related to the cannabis industry and understand that the cases thus far have overwhelmingly found against the cannabis businesses. The CPA needs to continually read about new tax cases and other current events related to the industry. There are many CPE courses available to CPAs that are specific to the cannabis industry. The CPA should also consult the AICPA, their state society of CPAs, or look to more mature cannabis states for learning opportunities. For example, the Colorado Society of Certified Public Accountants holds an annual symposium for CPAs interested in the cannabis industry. In addition, the accountant may want to interview a CPA from a more mature state to get advice on how to competently proceed and minimize risk.\n<\/p><p>CPAs must also remember that the nuances of accounting for the cannabis industry go beyond Tax Code Section 280E. For example, in newer cannabis states, the CRB may not be able to find a bank account. The CPA needs to be prepared to guide the CRB in establishing effective internal controls that will enable the CRB to safeguard inventory and cash, comply with all state and local regulations, and ensure reliable financial record-keeping of mostly cash transactions. The required internal controls will include, but go far beyond, physical controls such as cameras, vaults, and locks as required by regulations. State regulations also require effective and detailed inventory tracking, usually from seed-to-sale systems. Internal controls will also need to be sufficient to not only safeguard cash and cannabis inventory from theft but also ensure accurate revenue and expense records in sufficient detail for the CPA to properly complete tax returns at the federal, state, and local levels. Once banking is available to the CRB, a new set of reporting requirements will need to be met to maintain the account. Many CRBs will look to their CPA for help with these internal controls and reporting requirements.\n<\/p><p>Only one of the CPA participants from this study provides audit or other attest services to the cannabis industry. In general, there are many CPAs who do not provide any audit or attest services to any industry due to litigation risks. Some states, like Colorado, may at their discretion require a financial statement audit of a CRB by a licensed CPA. In addition, as the cannabis industry evolves, there may be additional funding opportunities that would put cannabis companies on the securities market, as has already taken place in Canada. This would cause these cannabis companies to require annual audited financial statements from a licensed CPA. While this study found there are CPAs that specialize in providing\u2014or are at least willing to provide\u2014accounting and tax services to the cannabis industry, there are not many firms that will do an audit for the industry. This presents additional opportunity for CPAs in the cannabis marketplace.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Limitations\">Limitations<\/span><\/h3>\n<p>This study has several limitations, including limited external validity and potential selection bias. The views of the interviewees may not necessarily generalize to all CRB owners or CPAs in these two states or other states. However, repetition of themes indicating data saturation is a good indication that the responses would be similar if more CRB owners or CBAs were interviewed. Finding participants for the study was difficult as many of those invited declined to participate. The convenience sample drawn could have introduced some level of selection bias into the study. Those who declined participation may have had different views of the industry than those that agreed to participate. In addition, interview responses may be influenced by the interviewer\u2019s presence and some participants may not be \u201cequally articulate and perceptive.\u201d<sup id=\"rdp-ebb-cite_ref-CreswellResearch18_55-5\" class=\"reference\"><a href=\"#cite_note-CreswellResearch18-55\">[55]<\/a><\/sup> While questions were asked in a neutral tone and data was analyzed without intentional bias, unconscious bias could be present in the study.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Recommendations_for_future_research\">Recommendations for future research<\/span><\/h3>\n<p>This study could be expanded to additional states to broaden this field of knowledge. Interviews of participants from states that are newer to the cannabis marketplace may result in findings that are different from the findings of this study of two mature cannabis states. A future study could compare the findings from mature states and developing states. Another interesting follow up study could investigate organization stigma issues for CPA firms working in the cannabis industry.\n<\/p><p>Quantitative studies could be designed to obtain data from a larger sample size than is possible with a qualitative study. For example, CPAs could be surveyed to determine their level of agreement with the theory identified herein that the greatest risk in CPAs providing services to CRBs is the risk of federal prosecution. CRB owners could be surveyed to determine their level of agreement with the theory that they need CPAs who have in-depth knowledge of the industry and would be able to perform as an outsourced CFO to guide their business into the future. Another quantitative study could be conducted to further explore the motivational or ethical factors as to why CPAs choose to serve this industry or not. A follow-up quantitative study has already begun in which CPAs have been surveyed to determine their level of agreement with the risks identified in this project and whether they perceive any ethical issues in working with the industry.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The cannabis industry remains in limbo with state laws in conflict with federal law. The findings of this study indicate that while many CPAs will not serve the cannabis industry, there are competent and knowledgeable CPAs who will. CRB owners need to carefully consider the industry knowledge and experience of a potential CPA prior to engaging them.\n<\/p><p>For CPAs that are considering accepting CRB clients, this study showed that there are risks that should be weighed, such as the risk of federal prosecution and potential loss of the CPA license. Next, the CPA must commit to acquiring and maintaining substantial specialized knowledge related to Tax Code Section 280E, internal controls for a cash-only or cash-intensive business, and the workings of the cannabis industry under the current regulatory conditions. As cannabis legalization continues to expand to new states, the need for CPAs to serve the industry continues to grow. This creates considerable business opportunity for CPAs who are willing to bear the related risk.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations.2C_acronyms.2C_and_initialisms\">Abbreviations, acronyms, and initialisms<\/span><\/h2>\n<p><b>AICPA<\/b>: American Institute of Certified Public Accountants\n<\/p><p><b>BOA<\/b>: Board of accountancy\n<\/p><p><b>CBD<\/b>: Cannabidiol\n<\/p><p><b>CFO<\/b>: Chief financial officer\n<\/p><p><b>CPA<\/b>: Certified public accountant\n<\/p><p><b>CRB<\/b>: Cannabis-related business\n<\/p><p><b>CSA<\/b>: Controlled Substances Act\n<\/p><p><b>IRB<\/b>: Institutional review board \n<\/p><p><b>THC<\/b>: Tetrahydrocannabinol \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_information\">Supplementary information<\/span><\/h2>\n<ul><li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1186%2Fs42238-020-00049-7\/MediaObjects\/42238_2020_49_MOESM1_ESM.docx\" target=\"_blank\">Appendix A<\/a><\/li>\n<li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1186%2Fs42238-020-00049-7\/MediaObjects\/42238_2020_49_MOESM2_ESM.docx\" target=\"_blank\">Appendix B<\/a><\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The primary author acknowledges her doctoral committee from California Southern University for their guidance on this research. The committee included Dr. Stephanie Hoon, Dr. Bruce Gillies, Dr. Mitchell Miller, Dr. Michael Ewald, and Dr. Jennifer Newmann. The author also acknowledges Dr. Richard C. Ott, Associate Professor of Statistics at Colorado Mesa University, for his assistance with the statistical analysis.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>There was one author. The primary author\u2019s doctoral committee from California Southern University is acknowledged. The author(s) read and approved the final manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Ethics_approval_and_consent_to_participate\">Ethics approval and consent to participate<\/span><\/h3>\n<p>IRB approval was obtained from California Southern University prior to data collection. All participants signed a consent to participate.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>The author received no funding for the research.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Availability_of_data_and_materials\">Availability of data and materials<\/span><\/h3>\n<p>Interview transcripts and manual notes as well as Excel files summarizing data were maintained.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The author has stated no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-NCSLState20-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NCSLState20_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-NCSLState20_1-1\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Conference on State Legislatures (September 2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ncsl.org\/research\/health\/state-medical-marijuana-laws.aspx\" target=\"_blank\">\"State medical Marijuana Laws\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ncsl.org\/research\/health\/state-medical-marijuana-laws.aspx\" target=\"_blank\">https:\/\/www.ncsl.org\/research\/health\/state-medical-marijuana-laws.aspx<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 22 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=State+medical+Marijuana+Laws&rft.atitle=&rft.aulast=National+Conference+on+State+Legislatures&rft.au=National+Conference+on+State+Legislatures&rft.date=September+2020&rft_id=https%3A%2F%2Fwww.ncsl.org%2Fresearch%2Fhealth%2Fstate-medical-marijuana-laws.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CampbellPot12-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CampbellPot12_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Campbell, G. (2012). <i>Pot, Inc.: Inside Medical Marijuana, America's Most Outlaw Industry<\/i>. Sterling. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781402779251.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Pot%2C+Inc.%3A+Inside+Medical+Marijuana%2C+America%27s+Most+Outlaw+Industry&rft.aulast=Campbell%2C+G.&rft.au=Campbell%2C+G.&rft.date=2012&rft.pub=Sterling&rft.isbn=9781402779251&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DOJControlled-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DOJControlled_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.deadiversion.usdoj.gov\/21cfr\/21usc\/811.htm\" target=\"_blank\">\"Title 21 United States Code (USC) Controlled Substances Act\"<\/a>. <i>Diversion Control Division<\/i>. U.S. Department of Justice<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.deadiversion.usdoj.gov\/21cfr\/21usc\/811.htm\" target=\"_blank\">https:\/\/www.deadiversion.usdoj.gov\/21cfr\/21usc\/811.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Title+21+United+States+Code+%28USC%29+Controlled+Substances+Act&rft.atitle=Diversion+Control+Division&rft.pub=U.S.+Department+of+Justice&rft_id=https%3A%2F%2Fwww.deadiversion.usdoj.gov%2F21cfr%2F21usc%2F811.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CongressHR1158-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CongressHR1158_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1158\/text\" target=\"_blank\">\"H.R.1158 - Consolidated Appropriations Act, 2020\"<\/a>. <i>Congress.gov<\/i>. Library of Congress. 20 December 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1158\/text\" target=\"_blank\">https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1158\/text<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=H.R.1158+-+Consolidated+Appropriations+Act%2C+2020&rft.atitle=Congress.gov&rft.date=20+December+2019&rft.pub=Library+of+Congress&rft_id=https%3A%2F%2Fwww.congress.gov%2Fbill%2F116th-congress%2Fhouse-bill%2F1158%2Ftext&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CongressHR2-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CongressHR2_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.congress.gov\/bill\/115th-congress\/house-bill\/2\" target=\"_blank\">\"H.R.2 - Agriculture Improvement Act of 2018\"<\/a>. <i>Congress.gov<\/i>. Library of Congress. 20 December 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.congress.gov\/bill\/115th-congress\/house-bill\/2\" target=\"_blank\">https:\/\/www.congress.gov\/bill\/115th-congress\/house-bill\/2<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=H.R.2+-+Agriculture+Improvement+Act+of+2018&rft.atitle=Congress.gov&rft.date=20+December+2018&rft.pub=Library+of+Congress&rft_id=https%3A%2F%2Fwww.congress.gov%2Fbill%2F115th-congress%2Fhouse-bill%2F2&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-USFDAHempProd19-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-USFDAHempProd19_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Abernethy, A. (25 July 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.fda.gov\/news-events\/congressional-testimony\/hemp-production-and-2018-farm-bill-07252019\" target=\"_blank\">\"Testimony: Hemp Production and the 2018 Farm Bill\"<\/a>. U.S. Food and Drug Administration<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.fda.gov\/news-events\/congressional-testimony\/hemp-production-and-2018-farm-bill-07252019\" target=\"_blank\">https:\/\/www.fda.gov\/news-events\/congressional-testimony\/hemp-production-and-2018-farm-bill-07252019<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Testimony%3A+Hemp+Production+and+the+2018+Farm+Bill&rft.atitle=&rft.aulast=Abernethy%2C+A.&rft.au=Abernethy%2C+A.&rft.date=25+July+2019&rft.pub=U.S.+Food+and+Drug+Administration&rft_id=https%3A%2F%2Fwww.fda.gov%2Fnews-events%2Fcongressional-testimony%2Fhemp-production-and-2018-farm-bill-07252019&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorAnAnal16-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TaylorAnAnal16_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-TaylorAnAnal16_7-1\">7.1<\/a><\/sup> <sup><a href=\"#cite_ref-TaylorAnAnal16_7-2\">7.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, K.; Bunker, R.B.; Johnson, L.R. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.abacademies.org\/journals\/month-november-year-2016-vol-19-issue-2-journal-jleri-past-issue.html\" target=\"_blank\">\"An analysis of the accounting and financial effects of inconsistent state and federal laws in the recreational cannabis industry\"<\/a>. <i>Journal of Legal, Ethical and Regulatory Issues<\/i> <b>19<\/b> (2): 11\u201325<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.abacademies.org\/journals\/month-november-year-2016-vol-19-issue-2-journal-jleri-past-issue.html\" target=\"_blank\">https:\/\/www.abacademies.org\/journals\/month-november-year-2016-vol-19-issue-2-journal-jleri-past-issue.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+analysis+of+the+accounting+and+financial+effects+of+inconsistent+state+and+federal+laws+in+the+recreational+cannabis+industry&rft.jtitle=Journal+of+Legal%2C+Ethical+and+Regulatory+Issues&rft.aulast=Taylor%2C+K.%3B+Bunker%2C+R.B.%3B+Johnson%2C+L.R.+et+al.&rft.au=Taylor%2C+K.%3B+Bunker%2C+R.B.%3B+Johnson%2C+L.R.+et+al.&rft.date=2016&rft.volume=19&rft.issue=2&rft.pages=11%E2%80%9325&rft_id=https%3A%2F%2Fwww.abacademies.org%2Fjournals%2Fmonth-november-year-2016-vol-19-issue-2-journal-jleri-past-issue.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AICPAAnIssue19-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AICPAAnIssue19_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAAnIssue19_8-1\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAAnIssue19_8-2\">8.2<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAAnIssue19_8-3\">8.3<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAAnIssue19_8-4\">8.4<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAAnIssue19_8-5\">8.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">AICPA (14 January 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.aicpa.org\/advocacy\/state\/downloadabledocuments\/marijuanacpasissuebrief.pdf\" target=\"_blank\">\"An issue brief on state marijuana laws and the CPA profession\"<\/a> (PDF). AICPA<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.aicpa.org\/advocacy\/state\/downloadabledocuments\/marijuanacpasissuebrief.pdf\" target=\"_blank\">https:\/\/www.aicpa.org\/advocacy\/state\/downloadabledocuments\/marijuanacpasissuebrief.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=An+issue+brief+on+state+marijuana+laws+and+the+CPA+profession&rft.atitle=&rft.aulast=AICPA&rft.au=AICPA&rft.date=14+January+2019&rft.pub=AICPA&rft_id=https%3A%2F%2Fwww.aicpa.org%2Fadvocacy%2Fstate%2Fdownloadabledocuments%2Fmarijuanacpasissuebrief.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GottliebInsurance16-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GottliebInsurance16_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gottlieb, L.; Munson, M. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.bpmlaw.com\/wp-content\/uploads\/2012\/04\/BRF_v046n01_Fall2016_Gottlieb-Munson-2.pdf\" target=\"_blank\">\"Insurance Issues Raised by the Legalization of Recreational Marijuana\"<\/a> (PDF). <i>The Brief<\/i> <b>46<\/b> (1): 34\u201339<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.bpmlaw.com\/wp-content\/uploads\/2012\/04\/BRF_v046n01_Fall2016_Gottlieb-Munson-2.pdf\" target=\"_blank\">http:\/\/www.bpmlaw.com\/wp-content\/uploads\/2012\/04\/BRF_v046n01_Fall2016_Gottlieb-Munson-2.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Insurance+Issues+Raised+by+the+Legalization+of+Recreational+Marijuana&rft.jtitle=The+Brief&rft.aulast=Gottlieb%2C+L.%3B+Munson%2C+M.&rft.au=Gottlieb%2C+L.%3B+Munson%2C+M.&rft.date=2016&rft.volume=46&rft.issue=1&rft.pages=34%E2%80%9339&rft_id=http%3A%2F%2Fwww.bpmlaw.com%2Fwp-content%2Fuploads%2F2012%2F04%2FBRF_v046n01_Fall2016_Gottlieb-Munson-2.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HouseUSC280E-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HouseUSC280E_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/uscode.house.gov\/view.xhtml?hl=false&edition=prelim&req=granuleid%3AUSC-prelim-title26-section280E\" target=\"_blank\">\"26 USC 280E: Expenditures in connection with the illegal sale of drugs\"<\/a>. <i>House.gov<\/i>. 03 September 1982<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/uscode.house.gov\/view.xhtml?hl=false&edition=prelim&req=granuleid%3AUSC-prelim-title26-section280E\" target=\"_blank\">https:\/\/uscode.house.gov\/view.xhtml?hl=false&edition=prelim&req=granuleid%3AUSC-prelim-title26-section280E<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=26+USC+280E%3A+Expenditures+in+connection+with+the+illegal+sale+of+drugs&rft.atitle=House.gov&rft.date=03+September+1982&rft_id=https%3A%2F%2Fuscode.house.gov%2Fview.xhtml%3Fhl%3Dfalse%26edition%3Dprelim%26req%3Dgranuleid%253AUSC-prelim-title26-section280E&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlsharaihaSmoking17-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlsharaihaSmoking17_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alsharaiha, M.N. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.utoledo.edu\/law\/studentlife\/lawreview\/volumes\/v48n2.html\" target=\"_blank\">\"Smoking Out the Criminals: How Federal Tax Policy Can Reduce Illegal Drug Crime by Supporting the Legal Marijuana Industry\"<\/a>. <i>University of Toledo Law Review<\/i> <b>48<\/b> (2): 319\u201336<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.utoledo.edu\/law\/studentlife\/lawreview\/volumes\/v48n2.html\" target=\"_blank\">https:\/\/www.utoledo.edu\/law\/studentlife\/lawreview\/volumes\/v48n2.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Smoking+Out+the+Criminals%3A+How+Federal+Tax+Policy+Can+Reduce+Illegal+Drug+Crime+by+Supporting+the+Legal+Marijuana+Industry&rft.jtitle=University+of+Toledo+Law+Review&rft.aulast=Alsharaiha%2C+M.N.&rft.au=Alsharaiha%2C+M.N.&rft.date=2017&rft.volume=48&rft.issue=2&rft.pages=319%E2%80%9336&rft_id=https%3A%2F%2Fwww.utoledo.edu%2Flaw%2Fstudentlife%2Flawreview%2Fvolumes%2Fv48n2.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Taillon280E18-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Taillon280E18_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taillon, B.; Cornelius, J. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cocpa.org\/product\/2018-marijuana-business-symposium\/\" target=\"_blank\">\"280E and the Value of Experienced CPAs\"<\/a>. <i>2018 Marijuana Business Symposium<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cocpa.org\/product\/2018-marijuana-business-symposium\/\" target=\"_blank\">https:\/\/www.cocpa.org\/product\/2018-marijuana-business-symposium\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=280E+and+the+Value+of+Experienced+CPAs&rft.jtitle=2018+Marijuana+Business+Symposium&rft.aulast=Taillon%2C+B.%3B+Cornelius%2C+J.&rft.au=Taillon%2C+B.%3B+Cornelius%2C+J.&rft.date=2018&rft_id=https%3A%2F%2Fwww.cocpa.org%2Fproduct%2F2018-marijuana-business-symposium%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DoJRICO-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DoJRICO_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.justice.gov\/archives\/jm\/criminal-resource-manual-956-rico-prosecutions-18-usc-1961-68\" target=\"_blank\">\"956. RICO PROSECUTIONS\u201418 U.S.C. \u00a7\u00a7 1961-68\"<\/a>. <i>U.S. Department of Justice Archives<\/i>. U.S. Department of Justice. 21 January 2020<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.justice.gov\/archives\/jm\/criminal-resource-manual-956-rico-prosecutions-18-usc-1961-68\" target=\"_blank\">https:\/\/www.justice.gov\/archives\/jm\/criminal-resource-manual-956-rico-prosecutions-18-usc-1961-68<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=956.+RICO+PROSECUTIONS%E2%80%9418+U.S.C.+%C2%A7%C2%A7+1961-68&rft.atitle=U.S.+Department+of+Justice+Archives&rft.date=21+January+2020&rft.pub=U.S.+Department+of+Justice&rft_id=https%3A%2F%2Fwww.justice.gov%2Farchives%2Fjm%2Fcriminal-resource-manual-956-rico-prosecutions-18-usc-1961-68&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DOJCriminalRICOPDF09-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DOJCriminalRICOPDF09_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Marine, F.J.; Crow, D.E.; Dalton, R.C. et al. (October 2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.justice.gov\/sites\/default\/files\/usao\/legacy\/2014\/10\/17\/rico.pdf\" target=\"_blank\">\"Criminal RICO: 18 U.S.C. \u00a7\u00a7 1961-1968 - A Manual for Federal Prosecutors, Fifth Revised Edition\"<\/a> (PDF). U.S. Department of Justice<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.justice.gov\/sites\/default\/files\/usao\/legacy\/2014\/10\/17\/rico.pdf\" target=\"_blank\">https:\/\/www.justice.gov\/sites\/default\/files\/usao\/legacy\/2014\/10\/17\/rico.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 06 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Criminal+RICO%3A+18+U.S.C.+%C2%A7%C2%A7+1961-1968+-+A+Manual+for+Federal+Prosecutors%2C+Fifth+Revised+Edition&rft.atitle=&rft.aulast=Marine%2C+F.J.%3B+Crow%2C+D.E.%3B+Dalton%2C+R.C.+et+al.&rft.au=Marine%2C+F.J.%3B+Crow%2C+D.E.%3B+Dalton%2C+R.C.+et+al.&rft.date=October+2009&rft.pub=U.S.+Department+of+Justice&rft_id=https%3A%2F%2Fwww.justice.gov%2Fsites%2Fdefault%2Ffiles%2Fusao%2Flegacy%2F2014%2F10%2F17%2Frico.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ReinhartUpIn16-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ReinhartUpIn16_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Reinhart, B.E. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.floridabar.org\/the-florida-bar-journal\/up-in-smoke-or-down-in-flames-a-florida-lawyers-legal-and-ethical-risks-in-advising-a-marijuana-industry-client\/\" target=\"_blank\">\"Up in Smoke or Down in Flames? A Florida Lawyer's Legal and Ethical Risks in Advising a Marijuana Industry Client\"<\/a>. <i>Florida Bar Journal<\/i> <b>90<\/b> (3): 20\u201328<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.floridabar.org\/the-florida-bar-journal\/up-in-smoke-or-down-in-flames-a-florida-lawyers-legal-and-ethical-risks-in-advising-a-marijuana-industry-client\/\" target=\"_blank\">https:\/\/www.floridabar.org\/the-florida-bar-journal\/up-in-smoke-or-down-in-flames-a-florida-lawyers-legal-and-ethical-risks-in-advising-a-marijuana-industry-client\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Up+in+Smoke+or+Down+in+Flames%3F+A+Florida+Lawyer%27s+Legal+and+Ethical+Risks+in+Advising+a+Marijuana+Industry+Client&rft.jtitle=Florida+Bar+Journal&rft.aulast=Reinhart%2C+B.E.&rft.au=Reinhart%2C+B.E.&rft.date=2016&rft.volume=90&rft.issue=3&rft.pages=20%E2%80%9328&rft_id=https%3A%2F%2Fwww.floridabar.org%2Fthe-florida-bar-journal%2Fup-in-smoke-or-down-in-flames-a-florida-lawyers-legal-and-ethical-risks-in-advising-a-marijuana-industry-client%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SandersToWeed15-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SandersToWeed15_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-SandersToWeed15_16-1\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sanders, R.C. (2015). \"To weed or not to weed? The Colorado quandary of legitimate cannabis businesses and the financial institutions who are unable to serve them\". <i>The Penn State Law Review<\/i> (Summer).<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=To+weed+or+not+to+weed%3F+The+Colorado+quandary+of+legitimate+cannabis+businesses+and+the+financial+institutions+who+are+unable+to+serve+them&rft.jtitle=The+Penn+State+Law+Review&rft.aulast=Sanders%2C+R.C.&rft.au=Sanders%2C+R.C.&rft.date=2015&rft.issue=Summer&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FCENAbout-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FCENAbout_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Financial Crimes Enforcement Network. \"What we do\". FINCEN.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+we+do&rft.atitle=&rft.aulast=Financial+Crimes+Enforcement+Network&rft.au=Financial+Crimes+Enforcement+Network&rft.pub=FINCEN&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HouserHigh15-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HouserHigh15_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Houser, K.A.; Rosacker, R.E. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.iabpad.com\/high-times-a-history-of-marijuana-laws-in-the-united-states\/\" target=\"_blank\">\"High times: A history of cannabis laws in the United States\"<\/a>. <i>International Journal of Business and Public Administration<\/i> <b>12<\/b> (2)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.iabpad.com\/high-times-a-history-of-marijuana-laws-in-the-united-states\/\" target=\"_blank\">http:\/\/www.iabpad.com\/high-times-a-history-of-marijuana-laws-in-the-united-states\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=High+times%3A+A+history+of+cannabis+laws+in+the+United+States&rft.jtitle=International+Journal+of+Business+and+Public+Administration&rft.aulast=Houser%2C+K.A.%3B+Rosacker%2C+R.E.&rft.au=Houser%2C+K.A.%3B+Rosacker%2C+R.E.&rft.date=2015&rft.volume=12&rft.issue=2&rft_id=http%3A%2F%2Fwww.iabpad.com%2Fhigh-times-a-history-of-marijuana-laws-in-the-united-states%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Shu-AcquayeTheUnint16-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Shu-AcquayeTheUnint16_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shu-Acquaye, F. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/engagedscholarship.csuohio.edu\/clevstlrev\/vol64\/iss2\/10\" target=\"_blank\">\"The Unintended Consequence to Legalizing Marijuana Use: The Banking Conondrum\"<\/a>. <i>Cleveland State Law Review<\/i> <b>64<\/b> (2): 315<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/engagedscholarship.csuohio.edu\/clevstlrev\/vol64\/iss2\/10\" target=\"_blank\">https:\/\/engagedscholarship.csuohio.edu\/clevstlrev\/vol64\/iss2\/10<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Unintended+Consequence+to+Legalizing+Marijuana+Use%3A+The+Banking+Conondrum&rft.jtitle=Cleveland+State+Law+Review&rft.aulast=Shu-Acquaye%2C+F.&rft.au=Shu-Acquaye%2C+F.&rft.date=2016&rft.volume=64&rft.issue=2&rft.pages=315&rft_id=https%3A%2F%2Fengagedscholarship.csuohio.edu%2Fclevstlrev%2Fvol64%2Fiss2%2F10&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DraytonTheGreat17-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DraytonTheGreat17_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dayton, M.R. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.marijuanaventure.com\/product\/mv-march-2017\/\" target=\"_blank\">\"The great unknown: Trump\u2019s impact on cannabis banking\"<\/a>. <i>Marijuana Venture<\/i> (3): 142\u20136<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.marijuanaventure.com\/product\/mv-march-2017\/\" target=\"_blank\">https:\/\/www.marijuanaventure.com\/product\/mv-march-2017\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+great+unknown%3A+Trump%E2%80%99s+impact+on+cannabis+banking&rft.jtitle=Marijuana+Venture&rft.aulast=Dayton%2C+M.R.&rft.au=Dayton%2C+M.R.&rft.date=2017&rft.issue=3&rft.pages=142%E2%80%936&rft_id=https%3A%2F%2Fwww.marijuanaventure.com%2Fproduct%2Fmv-march-2017%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FINCENMarij20-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FINCENMarij20_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Financial Crimes Enforcement Network (30 June 2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.fincen.gov\/sites\/default\/files\/shared\/508_295174_MJ%20Banking%20Update%203rd%20QTR%20FY2020_Public_508%20compliant.pdf\" target=\"_blank\">\"Marijuana Banking Update\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.fincen.gov\/sites\/default\/files\/shared\/508_295174_MJ%20Banking%20Update%203rd%20QTR%20FY2020_Public_508%20compliant.pdf\" target=\"_blank\">https:\/\/www.fincen.gov\/sites\/default\/files\/shared\/508_295174_MJ%20Banking%20Update%203rd%20QTR%20FY2020_Public_508%20compliant.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Marijuana+Banking+Update&rft.atitle=&rft.aulast=Financial+Crimes+Enforcement+Network&rft.au=Financial+Crimes+Enforcement+Network&rft.date=30+June+2020&rft_id=https%3A%2F%2Fwww.fincen.gov%2Fsites%2Fdefault%2Ffiles%2Fshared%2F508_295174_MJ%2520Banking%2520Update%25203rd%2520QTR%2520FY2020_Public_508%2520compliant.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CongressHR1588-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CongressHR1588_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1588\" target=\"_blank\">\"H.R.1588 - Ending Federal Marijuana Prohibition Act of 2019\"<\/a>. <i>Congress.gov<\/i>. Library of Congress. 08 April 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1588\" target=\"_blank\">https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1588<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=H.R.1588+-+Ending+Federal+Marijuana+Prohibition+Act+of+2019&rft.atitle=Congress.gov&rft.date=08+April+2019&rft.pub=Library+of+Congress&rft_id=https%3A%2F%2Fwww.congress.gov%2Fbill%2F116th-congress%2Fhouse-bill%2F1588&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CongressHR1595-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CongressHR1595_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1595\" target=\"_blank\">\"H.R.1595 - Secure And Fair Enforcement Banking Act of 2019\"<\/a>. <i>Congress.gov<\/i>. Library of Congress. 26 September 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1595\" target=\"_blank\">https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/1595<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=H.R.1595+-+Secure+And+Fair+Enforcement+Banking+Act+of+2019&rft.atitle=Congress.gov&rft.date=26+September+2019&rft.pub=Library+of+Congress&rft_id=https%3A%2F%2Fwww.congress.gov%2Fbill%2F116th-congress%2Fhouse-bill%2F1595&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CongressHR2703-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CongressHR2703_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/2703\" target=\"_blank\">\"H.R.2703 - MAPLE Act of 2019\"<\/a>. <i>Congress.gov<\/i>. Library of Congress. 26 June 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/2703\" target=\"_blank\">https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/2703<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=H.R.2703+-+MAPLE+Act+of+2019&rft.atitle=Congress.gov&rft.date=26+June+2019&rft.pub=Library+of+Congress&rft_id=https%3A%2F%2Fwww.congress.gov%2Fbill%2F116th-congress%2Fhouse-bill%2F2703&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CongressHR3884-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CongressHR3884_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/3884\" target=\"_blank\">\"H.R.3884 - MORE Act of 2020\"<\/a>. <i>Congress.gov<\/i>. Library of Congress. 07 December 2020<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/3884\" target=\"_blank\">https:\/\/www.congress.gov\/bill\/116th-congress\/house-bill\/3884<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=H.R.3884+-+MORE+Act+of+2020&rft.atitle=Congress.gov&rft.date=07+December+2020&rft.pub=Library+of+Congress&rft_id=https%3A%2F%2Fwww.congress.gov%2Fbill%2F116th-congress%2Fhouse-bill%2F3884&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MadisonInvent18-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MadisonInvent18_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Madison, C.J. (01 August 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.thetaxadviser.com\/issues\/2018\/aug\/inventory-accounting-cannabis-businesses.html\" target=\"_blank\">\"Inventory accounting for cannabis businesses: Sec. 280E and the impact of tax reform\"<\/a>. <i>The Tax Advisor<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.thetaxadviser.com\/issues\/2018\/aug\/inventory-accounting-cannabis-businesses.html\" target=\"_blank\">https:\/\/www.thetaxadviser.com\/issues\/2018\/aug\/inventory-accounting-cannabis-businesses.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Inventory+accounting+for+cannabis+businesses%3A+Sec.+280E+and+the+impact+of+tax+reform&rft.atitle=The+Tax+Advisor&rft.aulast=Madison%2C+C.J.&rft.au=Madison%2C+C.J.&rft.date=01+August+2018&rft_id=https%3A%2F%2Fwww.thetaxadviser.com%2Fissues%2F2018%2Faug%2Finventory-accounting-cannabis-businesses.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CODeptRevMedical16-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CODeptRevMedical16_27-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-CODeptRevMedical16_27-1\">27.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Department of Revenue, Marijuana Enforcement Division (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sos.state.co.us\/CCR\/GenerateRulePdf.do?ruleVersionId=6821\" target=\"_blank\">\"Medical Marijuana Rules - 1 CCR 212 - Version ID 6821\"<\/a> (PDF). State of Colorado<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sos.state.co.us\/CCR\/GenerateRulePdf.do?ruleVersionId=6821\" target=\"_blank\">https:\/\/www.sos.state.co.us\/CCR\/GenerateRulePdf.do?ruleVersionId=6821<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Medical+Marijuana+Rules+-+1+CCR+212+-+Version+ID+6821&rft.atitle=&rft.aulast=Department+of+Revenue%2C+Marijuana+Enforcement+Division&rft.au=Department+of+Revenue%2C+Marijuana+Enforcement+Division&rft.date=2016&rft.pub=State+of+Colorado&rft_id=https%3A%2F%2Fwww.sos.state.co.us%2FCCR%2FGenerateRulePdf.do%3FruleVersionId%3D6821&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CODeptRevMedical17-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CODeptRevMedical17_28-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-CODeptRevMedical17_28-1\">28.1<\/a><\/sup> <sup><a href=\"#cite_ref-CODeptRevMedical17_28-2\">28.2<\/a><\/sup> <sup><a href=\"#cite_ref-CODeptRevMedical17_28-3\">28.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Department of Revenue, Marijuana Enforcement Division (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sos.state.co.us\/CCR\/GenerateRulePdf.do?ruleVersionId=7094\" target=\"_blank\">\"Medical Marijuana Rules - 1 CCR 212 - Version ID 7094\"<\/a> (PDF). State of Colorado<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sos.state.co.us\/CCR\/GenerateRulePdf.do?ruleVersionId=7094\" target=\"_blank\">https:\/\/www.sos.state.co.us\/CCR\/GenerateRulePdf.do?ruleVersionId=7094<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Medical+Marijuana+Rules+-+1+CCR+212+-+Version+ID+7094&rft.atitle=&rft.aulast=Department+of+Revenue%2C+Marijuana+Enforcement+Division&rft.au=Department+of+Revenue%2C+Marijuana+Enforcement+Division&rft.date=2017&rft.pub=State+of+Colorado&rft_id=https%3A%2F%2Fwww.sos.state.co.us%2FCCR%2FGenerateRulePdf.do%3FruleVersionId%3D7094&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WSL314-55-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WSL314-55_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/apps.leg.wa.gov\/wac\/default.aspx?cite=314-55\" target=\"_blank\">\"Chapter 314-55 WAC, Marijuana Licenses, Application Process, Requirements, and Reporting\"<\/a>. <i>Washington State Legislature<\/i>. State of Washington. 2020<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/apps.leg.wa.gov\/wac\/default.aspx?cite=314-55\" target=\"_blank\">https:\/\/apps.leg.wa.gov\/wac\/default.aspx?cite=314-55<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+314-55+WAC%2C+Marijuana+Licenses%2C+Application+Process%2C+Requirements%2C+and+Reporting&rft.atitle=Washington+State+Legislature&rft.date=2020&rft.pub=State+of+Washington&rft_id=https%3A%2F%2Fapps.leg.wa.gov%2Fwac%2Fdefault.aspx%3Fcite%3D314-55&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FrazerInternal16-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FrazerInternal16_30-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-FrazerInternal16_30-1\">30.1<\/a><\/sup> <sup><a href=\"#cite_ref-FrazerInternal16_30-2\">30.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Frazer, L. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/articlegateway.com\/index.php\/JAF\/article\/view\/1047\" target=\"_blank\">\"Internal Control: Is it a Benefit or Fad to Small Companies? A Literature Dependency Perspective\"<\/a>. <i>Journal of Accounting and Finance<\/i> <b>16<\/b> (4): 149\u201361<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/articlegateway.com\/index.php\/JAF\/article\/view\/1047\" target=\"_blank\">https:\/\/articlegateway.com\/index.php\/JAF\/article\/view\/1047<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Internal+Control%3A+Is+it+a+Benefit+or+Fad+to+Small+Companies%3F+A+Literature+Dependency+Perspective&rft.jtitle=Journal+of+Accounting+and+Finance&rft.aulast=Frazer%2C+L.&rft.au=Frazer%2C+L.&rft.date=2016&rft.volume=16&rft.issue=4&rft.pages=149%E2%80%9361&rft_id=https%3A%2F%2Farticlegateway.com%2Findex.php%2FJAF%2Farticle%2Fview%2F1047&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhairulInternal16-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KhairulInternal16_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khairul, M.Z.; Anuar, N.; Ahmad, S.A.P.S. (2016). \"Internal controls and fraud \u2013 empirical evidence from oil and gas company\". <i>Journal of Financial Crime<\/i> <b>23<\/b> (4): 1154\u201368. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1108%2FJFC-04-2016-0021\" target=\"_blank\">10.1108\/JFC-04-2016-0021<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Internal+controls+and+fraud+%E2%80%93+empirical+evidence+from+oil+and+gas+company&rft.jtitle=Journal+of+Financial+Crime&rft.aulast=Khairul%2C+M.Z.%3B+Anuar%2C+N.%3B+Ahmad%2C+S.A.P.S.&rft.au=Khairul%2C+M.Z.%3B+Anuar%2C+N.%3B+Ahmad%2C+S.A.P.S.&rft.date=2016&rft.volume=23&rft.issue=4&rft.pages=1154%E2%80%9368&rft_id=info:doi\/10.1108%2FJFC-04-2016-0021&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HunzickerSolv18-32\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HunzickerSolv18_32-0\">32.0<\/a><\/sup> <sup><a href=\"#cite_ref-HunzickerSolv18_32-1\">32.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hunzicker, A. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.marijuanaventure.com\/product\/mv-august-2018\/\" target=\"_blank\">\"Solving financial mistakes made by cannabis businesses\"<\/a>. <i>Marijuana Venture<\/i> (8): 148\u201350<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.marijuanaventure.com\/product\/mv-august-2018\/\" target=\"_blank\">https:\/\/www.marijuanaventure.com\/product\/mv-august-2018\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Solving+financial+mistakes+made+by+cannabis+businesses&rft.jtitle=Marijuana+Venture&rft.aulast=Hunzicker%2C+A.&rft.au=Hunzicker%2C+A.&rft.date=2018&rft.issue=8&rft.pages=148%E2%80%9350&rft_id=https%3A%2F%2Fwww.marijuanaventure.com%2Fproduct%2Fmv-august-2018%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeeBigOpp18-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeeBigOpp18_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Lee, D. (01 August 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.accountingtoday.com\/news\/big-opportunities-in-cannabis-clients-compliance?tag=00000157-6c2f-dfbe-a35f-fcaf52df0000\" target=\"_blank\">\"Big opportunities in cannabis clients, compliance\"<\/a>. <i>Accounting Today<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.accountingtoday.com\/news\/big-opportunities-in-cannabis-clients-compliance?tag=00000157-6c2f-dfbe-a35f-fcaf52df0000\" target=\"_blank\">https:\/\/www.accountingtoday.com\/news\/big-opportunities-in-cannabis-clients-compliance?tag=00000157-6c2f-dfbe-a35f-fcaf52df0000<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Big+opportunities+in+cannabis+clients%2C+compliance&rft.atitle=Accounting+Today&rft.aulast=Lee%2C+D.&rft.au=Lee%2C+D.&rft.date=01+August+2018&rft_id=https%3A%2F%2Fwww.accountingtoday.com%2Fnews%2Fbig-opportunities-in-cannabis-clients-compliance%3Ftag%3D00000157-6c2f-dfbe-a35f-fcaf52df0000&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KappProtect11-34\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KappProtect11_34-0\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-KappProtect11_34-1\">34.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kapp, L.A.; Heslop, G. (2011). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nysscpa.org\/news\/publications\/the-cpa-journal\/article-preview?ArticleID=10702\" target=\"_blank\">\"Protecting Small Businesses from Fraud\"<\/a>. <i>The CPA Journal<\/i> <b>81<\/b> (10): 62\u20137<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nysscpa.org\/news\/publications\/the-cpa-journal\/article-preview?ArticleID=10702\" target=\"_blank\">https:\/\/www.nysscpa.org\/news\/publications\/the-cpa-journal\/article-preview?ArticleID=10702<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Protecting+Small+Businesses+from+Fraud&rft.jtitle=The+CPA+Journal&rft.aulast=Kapp%2C+L.A.%3B+Heslop%2C+G.&rft.au=Kapp%2C+L.A.%3B+Heslop%2C+G.&rft.date=2011&rft.volume=81&rft.issue=10&rft.pages=62%E2%80%937&rft_id=https%3A%2F%2Fwww.nysscpa.org%2Fnews%2Fpublications%2Fthe-cpa-journal%2Farticle-preview%3FArticleID%3D10702&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IRSCash10-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IRSCash10_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.irs.gov\/businesses\/small-businesses-self-employed\/cash-intensive-businesses-audit-techniques-guide-table-of-contents\" target=\"_blank\">\"Cash Intensive Businesses Audit Techniques Guide - Table of Contents\"<\/a>. Internal Revenue Service. April 2010<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.irs.gov\/businesses\/small-businesses-self-employed\/cash-intensive-businesses-audit-techniques-guide-table-of-contents\" target=\"_blank\">https:\/\/www.irs.gov\/businesses\/small-businesses-self-employed\/cash-intensive-businesses-audit-techniques-guide-table-of-contents<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cash+Intensive+Businesses+Audit+Techniques+Guide+-+Table+of+Contents&rft.atitle=&rft.date=April+2010&rft.pub=Internal+Revenue+Service&rft_id=https%3A%2F%2Fwww.irs.gov%2Fbusinesses%2Fsmall-businesses-self-employed%2Fcash-intensive-businesses-audit-techniques-guide-table-of-contents&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IRSCash10Chpt1-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IRSCash10Chpt1_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.irs.gov\/pub\/irs-utl\/cashchapter1_210627.pdf\" target=\"_blank\">\"Cash Intensive Businesses Audit Techniques - Chapter 1: Introduction and Overview of the Cash Intensive Business\"<\/a> (PDF). Internal Revenue Service. April 2010<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.irs.gov\/pub\/irs-utl\/cashchapter1_210627.pdf\" target=\"_blank\">https:\/\/www.irs.gov\/pub\/irs-utl\/cashchapter1_210627.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cash+Intensive+Businesses+Audit+Techniques+-+Chapter+1%3A+Introduction+and+Overview+of+the+Cash+Intensive+Business&rft.atitle=&rft.date=April+2010&rft.pub=Internal+Revenue+Service&rft_id=https%3A%2F%2Fwww.irs.gov%2Fpub%2Firs-utl%2Fcashchapter1_210627.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SatnaliwalaDont18-37\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SatnaliwalaDont18_37-0\">37.0<\/a><\/sup> <sup><a href=\"#cite_ref-SatnaliwalaDont18_37-1\">37.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Satnaliwala, M. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/go.gale.com\/ps\/anonymous?id=GALE\" target=\"_blank\">\"Don't Overlook Physical Access: Internal auditors need to evaluate how facilities are protected as part of the organization's overall security efforts\"<\/a>. <i>Internal Auditor<\/i> <b>75<\/b> (5): 22\u20133<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/go.gale.com\/ps\/anonymous?id=GALE\" target=\"_blank\">https:\/\/go.gale.com\/ps\/anonymous?id=GALE<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Don%27t+Overlook+Physical+Access%3A+Internal+auditors+need+to+evaluate+how+facilities+are+protected+as+part+of+the+organization%27s+overall+security+efforts&rft.jtitle=Internal+Auditor&rft.aulast=Satnaliwala%2C+M.&rft.au=Satnaliwala%2C+M.&rft.date=2018&rft.volume=75&rft.issue=5&rft.pages=22%E2%80%933&rft_id=https%3A%2F%2Fgo.gale.com%2Fps%2Fanonymous%3Fid%3DGALE&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArensAudit17-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArensAudit17_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Arens, A.A.; Elder, R.J.; Beasley, M.S. et al. (2017). <i>Auditing and Assurance Services: An Integrated Approach<\/i> (16th ed.). Pearson. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780134065823.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Auditing+and+Assurance+Services%3A+An+Integrated+Approach&rft.aulast=Arens%2C+A.A.%3B+Elder%2C+R.J.%3B+Beasley%2C+M.S.+et+al.&rft.au=Arens%2C+A.A.%3B+Elder%2C+R.J.%3B+Beasley%2C+M.S.+et+al.&rft.date=2017&rft.edition=16th&rft.pub=Pearson&rft.isbn=9780134065823&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SimkinCore14-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SimkinCore14_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Simkins, M.G.; Norman, C.A.S.; Rose, J.G. (2014). <i>Core Concepts of Accounting Information Systems<\/i> (13th ed.). Wiley. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781119033288.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Core+Concepts+of+Accounting+Information+Systems&rft.aulast=Simkins%2C+M.G.%3B+Norman%2C+C.A.S.%3B+Rose%2C+J.G.&rft.au=Simkins%2C+M.G.%3B+Norman%2C+C.A.S.%3B+Rose%2C+J.G.&rft.date=2014&rft.edition=13th&rft.pub=Wiley&rft.isbn=9781119033288&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AICPACode14-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AICPACode14_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">AICPA (15 December 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/pub.aicpa.org\/codeofconduct\/Ethics.aspx\" target=\"_blank\">\"Code of Professional Conduct\"<\/a>. AICPA<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/pub.aicpa.org\/codeofconduct\/Ethics.aspx\" target=\"_blank\">https:\/\/pub.aicpa.org\/codeofconduct\/Ethics.aspx<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Code+of+Professional+Conduct&rft.atitle=&rft.aulast=AICPA&rft.au=AICPA&rft.date=15+December+2014&rft.pub=AICPA&rft_id=https%3A%2F%2Fpub.aicpa.org%2Fcodeofconduct%2FEthics.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SatterlundCPA18-41\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SatterlundCPA18_41-0\">41.0<\/a><\/sup> <sup><a href=\"#cite_ref-SatterlundCPA18_41-1\">41.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Satterlund, C. (15 March 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/acb.wa.gov\/cpas-and-cannabis\" target=\"_blank\">\"CPAs and Cannabis\"<\/a>. <i>Washington State Board of Accountancy<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/acb.wa.gov\/cpas-and-cannabis\" target=\"_blank\">https:\/\/acb.wa.gov\/cpas-and-cannabis<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 September 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=CPAs+and+Cannabis&rft.atitle=Washington+State+Board+of+Accountancy&rft.aulast=Satterlund%2C+C.&rft.au=Satterlund%2C+C.&rft.date=15+March+2018&rft_id=https%3A%2F%2Facb.wa.gov%2Fcpas-and-cannabis&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AICPAProviding19-42\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AICPAProviding19_42-0\">42.0<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAProviding19_42-1\">42.1<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAProviding19_42-2\">42.2<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAProviding19_42-3\">42.3<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAProviding19_42-4\">42.4<\/a><\/sup> <sup><a href=\"#cite_ref-AICPAProviding19_42-5\">42.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">AICPA (January 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.aicpa.org\/advocacy\/state\/downloadabledocuments\/marijuana-state-board-positions.pdf\" target=\"_blank\">\"Providing services to businesses in the marijuana industry: A sample of current board positions\"<\/a> (PDF). AICPA<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.aicpa.org\/advocacy\/state\/downloadabledocuments\/marijuana-state-board-positions.pdf\" target=\"_blank\">https:\/\/www.aicpa.org\/advocacy\/state\/downloadabledocuments\/marijuana-state-board-positions.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 22 October 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Providing+services+to+businesses+in+the+marijuana+industry%3A+A+sample+of+current+board+positions&rft.atitle=&rft.aulast=AICPA&rft.au=AICPA&rft.date=January+2019&rft.pub=AICPA&rft_id=https%3A%2F%2Fwww.aicpa.org%2Fadvocacy%2Fstate%2Fdownloadabledocuments%2Fmarijuana-state-board-positions.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SternaLiab17-43\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SternaLiab17_43-0\">43.0<\/a><\/sup> <sup><a href=\"#cite_ref-SternaLiab17_43-1\">43.1<\/a><\/sup> <sup><a href=\"#cite_ref-SternaLiab17_43-2\">43.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sterna, S.; Wolfe, J. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cpajournal.com\/2017\/10\/19\/liability-risks-concerns-servicing-marijuana-businesses\/\" target=\"_blank\">\"Liability Risks and Other Concerns when Servicing Marijuana Businesses\"<\/a>. <i>The CPA Journal<\/i> <b>87<\/b> (10): 9\u201310<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cpajournal.com\/2017\/10\/19\/liability-risks-concerns-servicing-marijuana-businesses\/\" target=\"_blank\">https:\/\/www.cpajournal.com\/2017\/10\/19\/liability-risks-concerns-servicing-marijuana-businesses\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Liability+Risks+and+Other+Concerns+when+Servicing+Marijuana+Businesses&rft.jtitle=The+CPA+Journal&rft.aulast=Sterna%2C+S.%3B+Wolfe%2C+J.&rft.au=Sterna%2C+S.%3B+Wolfe%2C+J.&rft.date=2017&rft.volume=87&rft.issue=10&rft.pages=9%E2%80%9310&rft_id=https%3A%2F%2Fwww.cpajournal.com%2F2017%2F10%2F19%2Fliability-risks-concerns-servicing-marijuana-businesses%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OhanesianPrep18-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OhanesianPrep18_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Ohanesian, M.V. (01 March 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.thetaxadviser.com\/issues\/2018\/mar\/preparing-work-indirectly-cannabis-industry.html\" target=\"_blank\">\"Preparing to work indirectly with the cannabis industry\"<\/a>. <i>The Tax Adviser<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.thetaxadviser.com\/issues\/2018\/mar\/preparing-work-indirectly-cannabis-industry.html\" target=\"_blank\">https:\/\/www.thetaxadviser.com\/issues\/2018\/mar\/preparing-work-indirectly-cannabis-industry.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Preparing+to+work+indirectly+with+the+cannabis+industry&rft.atitle=The+Tax+Adviser&rft.aulast=Ohanesian%2C+M.V.&rft.au=Ohanesian%2C+M.V.&rft.date=01+March+2018&rft_id=https%3A%2F%2Fwww.thetaxadviser.com%2Fissues%2F2018%2Fmar%2Fpreparing-work-indirectly-cannabis-industry.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeversAGen08-45\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DeversAGen08_45-0\">45.0<\/a><\/sup> <sup><a href=\"#cite_ref-DeversAGen08_45-1\">45.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Devers, C.E.; Dewett, T.; Mishina, Y. et al. (2009). \"A General Theory of Organizational Stigma\". <i>Organization Science<\/i> <b>20<\/b> (1): 154\u2013171. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Forsc.1080.0367\" target=\"_blank\">10.1287\/orsc.1080.0367<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+General+Theory+of+Organizational+Stigma&rft.jtitle=Organization+Science&rft.aulast=Devers%2C+C.E.%3B+Dewett%2C+T.%3B+Mishina%2C+Y.+et+al.&rft.au=Devers%2C+C.E.%3B+Dewett%2C+T.%3B+Mishina%2C+Y.+et+al.&rft.date=2009&rft.volume=20&rft.issue=1&rft.pages=154%E2%80%93171&rft_id=info:doi\/10.1287%2Forsc.1080.0367&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HudsonNotWith09-46\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HudsonNotWith09_46-0\">46.0<\/a><\/sup> <sup><a href=\"#cite_ref-HudsonNotWith09_46-1\">46.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hudson, B.A.; Okhuysen, G.A. (2009). \"Not with a Ten-Foot Pole: Core Stigma, Stigma Transfer, and Improbable Persistence of Men's Bathhouses\". <i>Organization Science<\/i> <b>20<\/b> (1): 134\u201353. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Forsc.1080.0368\" target=\"_blank\">10.1287\/orsc.1080.0368<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Not+with+a+Ten-Foot+Pole%3A+Core+Stigma%2C+Stigma+Transfer%2C+and+Improbable+Persistence+of+Men%27s+Bathhouses&rft.jtitle=Organization+Science&rft.aulast=Hudson%2C+B.A.%3B+Okhuysen%2C+G.A.&rft.au=Hudson%2C+B.A.%3B+Okhuysen%2C+G.A.&rft.date=2009&rft.volume=20&rft.issue=1&rft.pages=134%E2%80%9353&rft_id=info:doi\/10.1287%2Forsc.1080.0368&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HampelHow16-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HampelHow16_47-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hampel, C.E.; Tracey, P. (2016). \"How Organizations Move from Stigma to Legitimacy: The Case of Cook\u2019s Travel Agency in Victorian Britain\". <i>Academy of Management Journal<\/i> <b>60<\/b> (6): 2175\u2013207. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5465%2Famj.2015.0365\" target=\"_blank\">10.5465\/amj.2015.0365<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+Organizations+Move+from+Stigma+to+Legitimacy%3A+The+Case+of+Cook%E2%80%99s+Travel+Agency+in+Victorian+Britain&rft.jtitle=Academy+of+Management+Journal&rft.aulast=Hampel%2C+C.E.%3B+Tracey%2C+P.&rft.au=Hampel%2C+C.E.%3B+Tracey%2C+P.&rft.date=2016&rft.volume=60&rft.issue=6&rft.pages=2175%E2%80%93207&rft_id=info:doi\/10.5465%2Famj.2015.0365&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GaetanoNYSSCPA17-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GaetanoNYSSCPA17_48-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gaetano, C. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nysscpa.org\/docs\/default-source\/trusted-professional-archives\/tp_jan_feb_17.pdf\" target=\"_blank\">\"NYSSCPA Marijuana Symposium addresses risks and benefits of the industry\"<\/a> (PDF). <i>The Trusted Professional<\/i> <b>20<\/b> (1): 1<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nysscpa.org\/docs\/default-source\/trusted-professional-archives\/tp_jan_feb_17.pdf\" target=\"_blank\">https:\/\/www.nysscpa.org\/docs\/default-source\/trusted-professional-archives\/tp_jan_feb_17.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=NYSSCPA+Marijuana+Symposium+addresses+risks+and+benefits+of+the+industry&rft.jtitle=The+Trusted+Professional&rft.aulast=Gaetano%2C+C.&rft.au=Gaetano%2C+C.&rft.date=2017&rft.volume=20&rft.issue=1&rft.pages=1&rft_id=https%3A%2F%2Fwww.nysscpa.org%2Fdocs%2Fdefault-source%2Ftrusted-professional-archives%2Ftp_jan_feb_17.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BelottoData18-49\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BelottoData18_49-0\">49.0<\/a><\/sup> <sup><a href=\"#cite_ref-BelottoData18_49-1\">49.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Belotto, M.J. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/nsuworks.nova.edu\/tqr\/vol23\/iss11\/2\" target=\"_blank\">\"Data Analysis Methods for Qualitative Research: Managing the Challenges of Coding, Interrater Reliability, and Thematic Analysis\"<\/a>. <i>The Qualitative Report<\/i> <b>23<\/b> (11): 2622-2633<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/nsuworks.nova.edu\/tqr\/vol23\/iss11\/2\" target=\"_blank\">https:\/\/nsuworks.nova.edu\/tqr\/vol23\/iss11\/2<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Analysis+Methods+for+Qualitative+Research%3A+Managing+the+Challenges+of+Coding%2C+Interrater+Reliability%2C+and+Thematic+Analysis&rft.jtitle=The+Qualitative+Report&rft.aulast=Belotto%2C+M.J.&rft.au=Belotto%2C+M.J.&rft.date=2018&rft.volume=23&rft.issue=11&rft.pages=2622-2633&rft_id=https%3A%2F%2Fnsuworks.nova.edu%2Ftqr%2Fvol23%2Fiss11%2F2&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AstalinQual13-50\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AstalinQual13_50-0\">50.0<\/a><\/sup> <sup><a href=\"#cite_ref-AstalinQual13_50-1\">50.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Astalin, P.K. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.i-scholar.in\/index.php\/ijssir\/article\/view\/43645\" target=\"_blank\">\"Qualitative Research Designs: A Conceptual Framework\"<\/a>. <i>International Journal of Social Sciences & Interdisciplinary Research<\/i> <b>2<\/b> (1): 118\u201324<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.i-scholar.in\/index.php\/ijssir\/article\/view\/43645\" target=\"_blank\">http:\/\/www.i-scholar.in\/index.php\/ijssir\/article\/view\/43645<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Qualitative+Research+Designs%3A+A+Conceptual+Framework&rft.jtitle=International+Journal+of+Social+Sciences+%26+Interdisciplinary+Research&rft.aulast=Astalin%2C+P.K.&rft.au=Astalin%2C+P.K.&rft.date=2013&rft.volume=2&rft.issue=1&rft.pages=118%E2%80%9324&rft_id=http%3A%2F%2Fwww.i-scholar.in%2Findex.php%2Fijssir%2Farticle%2Fview%2F43645&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Savin-BadenQual13-51\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Savin-BadenQual13_51-0\">51.0<\/a><\/sup> <sup><a href=\"#cite_ref-Savin-BadenQual13_51-1\">51.1<\/a><\/sup> <sup><a href=\"#cite_ref-Savin-BadenQual13_51-2\">51.2<\/a><\/sup> <sup><a href=\"#cite_ref-Savin-BadenQual13_51-3\">51.3<\/a><\/sup> <sup><a href=\"#cite_ref-Savin-BadenQual13_51-4\">51.4<\/a><\/sup> <sup><a href=\"#cite_ref-Savin-BadenQual13_51-5\">51.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Savin-Baden, M.; Major, C.H. (2013). <i>Qualitative Research: The Essential Guide to Theory and Practice<\/i>. Routledge. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780415674782.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Qualitative+Research%3A+The+Essential+Guide+to+Theory+and+Practice&rft.aulast=Savin-Baden%2C+M.%3B+Major%2C+C.H.&rft.au=Savin-Baden%2C+M.%3B+Major%2C+C.H.&rft.date=2013&rft.pub=Routledge&rft.isbn=9780415674782&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CooperBusi13-52\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CooperBusi13_52-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Cooper, D.R.; Schindler, P.S. (2013). <i>Business Research Methods<\/i> (12th ed.). McGraw-Hill Education. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780073521503.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Business+Research+Methods&rft.aulast=Cooper%2C+D.R.%3B+Schindler%2C+P.S.&rft.au=Cooper%2C+D.R.%3B+Schindler%2C+P.S.&rft.date=2013&rft.edition=12th&rft.pub=McGraw-Hill+Education&rft.isbn=9780073521503&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MerriamQual09-53\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MerriamQual09_53-0\">53.0<\/a><\/sup> <sup><a href=\"#cite_ref-MerriamQual09_53-1\">53.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Merriam, S.B. (2009). <i>Qualitative Research: A Guide to Design and Implementation<\/i> (3rd ed.). Jossey-Bass. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780470283547.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Qualitative+Research%3A+A+Guide+to+Design+and+Implementation&rft.aulast=Merriam%2C+S.B.&rft.au=Merriam%2C+S.B.&rft.date=2009&rft.edition=3rd&rft.pub=Jossey-Bass&rft.isbn=9780470283547&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AstrothFocus18-54\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AstrothFocus18_54-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Astroth, K.S. (2018). \"Focusing on the Fundamentals: Reading Qualitative Research with a Critical Eye\". <i>Nephrology Nursing Journal<\/i> <b>45<\/b> (4): 381\u201386. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30303648\" target=\"_blank\">30303648<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Focusing+on+the+Fundamentals%3A+Reading+Qualitative+Research+with+a+Critical+Eye&rft.jtitle=Nephrology+Nursing+Journal&rft.aulast=Astroth%2C+K.S.&rft.au=Astroth%2C+K.S.&rft.date=2018&rft.volume=45&rft.issue=4&rft.pages=381%E2%80%9386&rft_id=info:pmid\/30303648&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CreswellResearch18-55\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CreswellResearch18_55-0\">55.0<\/a><\/sup> <sup><a href=\"#cite_ref-CreswellResearch18_55-1\">55.1<\/a><\/sup> <sup><a href=\"#cite_ref-CreswellResearch18_55-2\">55.2<\/a><\/sup> <sup><a href=\"#cite_ref-CreswellResearch18_55-3\">55.3<\/a><\/sup> <sup><a href=\"#cite_ref-CreswellResearch18_55-4\">55.4<\/a><\/sup> <sup><a href=\"#cite_ref-CreswellResearch18_55-5\">55.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Creswell, J.W.; Creswell, J.D. (2018). <i>Research Design: Qualitative, Quantitative, and Mixed Methods Approaches<\/i> (5th ed.). SAGE Publications, Inc. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781506386706.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Research+Design%3A+Qualitative%2C+Quantitative%2C+and+Mixed+Methods+Approaches&rft.aulast=Creswell%2C+J.W.%3B+Creswell%2C+J.D.&rft.au=Creswell%2C+J.W.%3B+Creswell%2C+J.D.&rft.date=2018&rft.edition=5th&rft.pub=SAGE+Publications%2C+Inc&rft.isbn=9781506386706&rfr_id=info:sid\/en.wikipedia.org:Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; this version lists them in order of appearance, by design. Two of the original references for AICPA CPA docs had broken URLs; what is presumed to be the same documents\u2014although updated for 2019\u2014were found elsewhere on the AICPA site and used for this version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210429194055\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.831 seconds\nReal time usage: 0.861 seconds\nPreprocessor visited node count: 37708\/1000000\nPreprocessor generated node count: 42805\/1000000\nPost\u2010expand include size: 262473\/2097152 bytes\nTemplate argument size: 93132\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 773.088 1 - -total\n 87.04% 672.897 1 - Template:Reflist\n 72.53% 560.744 55 - Template:Citation\/core\n 33.98% 262.705 26 - Template:Cite_web\n 31.49% 243.452 22 - Template:Cite_journal\n 10.93% 84.472 7 - Template:Cite_book\n 5.03% 38.865 1 - Template:Infobox_journal_article\n 4.73% 36.531 1 - Template:Infobox\n 4.18% 32.340 69 - Template:Citation\/make_link\n 3.22% 24.871 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12404-0!*!*!!en!*!* and timestamp 20210429194054 and revision id 41830\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners\">https:\/\/www.limswiki.org\/index.php\/Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","a40f37542af50527bcd32bf00da69900_images":[],"a40f37542af50527bcd32bf00da69900_timestamp":1619725254,"74856edf0e6bde68458770a07660ece5":{"type":"chapter","title":"1. Cannabis analyses","key":"74856edf0e6bde68458770a07660ece5"}},"link":"https:\/\/www.limswiki.org\/index.php\/Book:LIMSjournal_-_Spring_2021","price_currency":"","price_amount":"","book_size":"","download_url":"https:\/\/www.limsforum.com?ebb_action=book_download&book_id=96683","language":"","cta_button_content":"","toc":[{"type":"chapter","name":"1. Cannabis analyses","id":"74856edf0e6bde68458770a07660ece5","children":[{"type":"article","name":"Accounting and the US cannabis industry: Federal financial regulations and the perspectives of certified public accountants and cannabis businesses owners (Owens-Ott 2020)","id":"a40f37542af50527bcd32bf00da69900","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Accounting_and_the_US_cannabis_industry:_Federal_financial_regulations_and_the_perspectives_of_certified_public_accountants_and_cannabis_businesses_owners"},{"type":"article","name":"Development of a gas-chromatographic method for simultaneous determination of cannabinoids and terpenes in hemp (Zeki\u010d and Kri\u017eman 2020)","id":"f48cdd8c7fc25a130c45d85871f3ab72","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Development_of_a_gas-chromatographic_method_for_simultaneous_determination_of_cannabinoids_and_terpenes_in_hemp"}]},{"type":"chapter","name":"2. Clinical informatics","id":"cc883ed5b173beeb241cab635b32933a","children":[{"type":"article","name":"Implement an international interoperable PHR by FHIR: A Taiwan innovative application (Lee et al. 2020)","id":"bd8eee413799c826d26140a4bd9d594e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Implement_an_international_interoperable_PHR_by_FHIR:_A_Taiwan_innovative_application"}]},{"type":"chapter","name":"3. Data science and management","id":"e8172473093328f6a7f1d59e580b48d1","children":[{"type":"article","name":"Kadi4Mat: A research data infrastructure for materials science (Brandt et al. 2021)","id":"7d7600ccadb42ccd7bdbe90737c9955d","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Kadi4Mat:_A_research_data_infrastructure_for_materials_science"},{"type":"article","name":"Named data networking for genomics data management and integrated workflows (Ogle et al. 2021)","id":"784b63abe742a2cc6286e50d625220ec","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Named_data_networking_for_genomics_data_management_and_integrated_workflows"},{"type":"article","name":"Making data and workflows findable for machines (Weigel et al. 2020)","id":"79a0829b6677ab7c03477c2b7c49428a","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Making_data_and_workflows_findable_for_machines"},{"type":"article","name":"Towards a risk catalog for data management plans (Weng and Thoben 2020)","id":"00bcc6c235846f18221f6a618b37957c","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Towards_a_risk_catalog_for_data_management_plans"}]},{"type":"chapter","name":"4. Laboratory and scientific informatics","id":"2936237187d01cfa2a1c44845b2a3795","children":[{"type":"article","name":"Digital transformation risk management in forensic science laboratories (Casey and Souvignet 2020)","id":"b18b6bb1bbee57c49590e7b3ba53097d","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Digital_transformation_risk_management_in_forensic_science_laboratories"},{"type":"article","name":"Timely delivery of laboratory efficiency information, Part I: Developing an interactive turnaround time dashboard at a high-volume laboratory (Cassim et al. 2020)","id":"d11707b0ec9b9843d1f6332c3f2b2930","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_I:_Developing_an_interactive_turnaround_time_dashboard_at_a_high-volume_laboratory"},{"type":"article","name":"Timely delivery of laboratory efficiency information, Part II: Assessing the impact of a turnaround time dashboard at a high-volume laboratory (Cassim et al. 2020)","id":"2897b25895d56e2e3f8922c52ab067ea","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Timely_delivery_of_laboratory_efficiency_information,_Part_II:_Assessing_the_impact_of_a_turnaround_time_dashboard_at_a_high-volume_laboratory"},{"type":"article","name":"Advanced engineering informatics: Philosophical and methodological foundations with examples from civil and construction engineering (Hartmann and Trappey 2020)","id":"0dd74698c3756484b9f52cadbcdc94dc","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Advanced_engineering_informatics:_Philosophical_and_methodological_foundations_with_examples_from_civil_and_construction_engineering"},{"type":"article","name":"Laboratory technology planning and management: The practice of laboratory systems engineering (Liscouski 2020)","id":"655f7d48a642e9b45533745af73f0d59","pageUrl":"https:\/\/www.limswiki.org\/index.php\/LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering"},{"type":"article","name":"Considerations in the automation of laboratory procedures (Liscouski 2021)","id":"e0147011cc1eb892e1a35e821657a6d9","pageUrl":"https:\/\/www.limswiki.org\/index.php\/LII:Considerations_in_the_Automation_of_Laboratory_Procedures"}]}],"settings":{"show_cover":"1","show_title":"1","show_subtitle":"0","show_full_title":"1","show_editor":"1","show_editor_pic":"1","show_publisher":"1","show_language":"1","show_size":"1","show_toc":"1","show_content_beneath_cover":"1","cta_button":"1","content_location":"1","toc_links":"disabled","log_in_msg":"<span><\/span> Please log in to read online.","cover_size":"medium"},"title_image":"https:\/\/s3.limsforum.com\/www.limsforum.com\/wp-content\/uploads\/Fig8_Liscouski_LabTechPlanMan20.png"}}
LIMSjournal - Spring 2021
Volume 7, Issue 1
Editor: Shawn Douglas
Publisher: LabLynx Press
Copyright LabLynx Inc. All rights reserved.