{"ID":97793,"post_author":"9412100","post_date":"2021-07-05 11:05:43","post_date_gmt":"0000-00-00 00:00:00","post_content":"","post_title":"LIMSjournal - Summer 2021","post_excerpt":"","post_status":"draft","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"","to_ping":"","pinged":"","post_modified":"2021-07-05 11:05:43","post_modified_gmt":"2021-07-05 15:05:43","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.limsforum.com\/?post_type=ebook&p=97793","menu_order":0,"post_type":"ebook","post_mime_type":"","comment_count":"0","filter":"","_ebook_metadata":{"enabled":"on","private":"0","guid":"18FC3EBC-C556-4340-9D0B-6349325AA628","title":"LIMSjournal - Summer 2021","subtitle":"Volume 7, Issue 2","cover_theme":"nico_21","cover_image":"https:\/\/www.limsforum.com\/wp-content\/plugins\/rdp-ebook-builder\/pl\/cover.php?cover_style=nico_21&subtitle=Volume+7%2C+Issue+2&editor=Shawn+Douglas&title=LIMSjournal+-+Summer+2021&title_image=https%3A%2F%2Fs3.limsforum.com%2Fwww.limsforum.com%2Fwp-content%2Fuploads%2FFig5_Jebali_JofInfoTelec2020_5-1.jpg&publisher=LabLynx+Press","editor":"Shawn Douglas","publisher":"LabLynx Press","author_id":"26","image_url":"","items":{"25073682c23be675cc4cd14be481b85b_type":"article","25073682c23be675cc4cd14be481b85b_title":"Informatics-driven quality improvement in the modern histology lab (Seifert et al. 2020)","25073682c23be675cc4cd14be481b85b_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab","25073682c23be675cc4cd14be481b85b_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Informatics-driven quality improvement in the modern histology lab\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nInformatics-driven quality improvement in the modern histology labJournal\n \nJAMIA OpenAuthor(s)\n \nSeifert, Robert P.; Casler, Vektra; Al Qaysi, Nada; Gothi, Shaileshbhai R.; Williams, Leah; Christensen, Patricia R.; Flax, Sherri; Chamala, SrikarAuthor affiliation(s)\n \nUniversity of Florida, UF Health Medical LaboratoriesPrimary contact\n \nschamala at ufl dot eduYear published\n \n2020Volume and issue\n \n3Page(s)\n \n4DOI\n \n10.1093\/jamiaopen\/ooaa066ISSN\n \n2574-2531Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/academic.oup.com\/jamiaopen\/article\/3\/4\/530\/6012900Download\n \nhttps:\/\/academic.oup.com\/jamiaopen\/article-pdf\/3\/4\/530\/36625866\/ooaa066.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Encountered challenges and tailored informatics solutions \n\n3.1 Histotechnologist status board \n3.2 Slide volume, compensation, and resident training \n3.3 Subspecialty-specific turnaround time investigation \n3.4 Delayed slide delivery \n3.5 Same-day immunohistochemistry order volume surge detection \n3.6 Specimen packing and paperwork deviation \n\n\n4 Conclusion \n5 Acknowledgements \n\n5.1 Author contributions \n5.2 Funding \n5.3 Conflict of interest \n\n\n6 References \n7 Notes \n\n\n\nAbstract \nLaboratory information systems (LISs) and data visualization techniques have untapped potential in anatomic pathology laboratories. The pre-built functionalities of an LIS do not address all the needs of a modern histology laboratory. For instance, \u201cgo live\u201d is not the end of LIS customization, only the beginning. After closely evaluating various histology lab workflows, we implemented several custom data analytics dashboards and additional LIS functionalities to monitor and address weaknesses. Herein, we present our experience with LIS and data-tracking solutions that improved trainee education, slide logistics, staffing and instrumentation lobbying, and task tracking. The latter was addressed through the creation of a novel \u201cstatus board\u201d akin to those seen in inpatient wards. These use-cases can benefit other histology laboratories.\nLay summary: Histopathology is the branch of medicine that involves the gross and microscopic examination of sampled tissue to help reach a diagnosis. Histology lab workflows incorporate a multitude of automated and manual steps that require close monitoring. Data visualization is a powerful tool that can illustrate workflow trends using a graphic representation of data. This manuscript describes data visualization and data analysis techniques leveraged to address frequent pitfalls in histology laboratory workflows. This led to multiple improvements in patient safety and quality of care. Our use-case solutions, presented here, can be adapted by other histology laboratories to overcome common laboratory challenges.\nKeywords: informatics, laboratory information system, quality improvement, pathology, workflow\n\nIntroduction \nThe microscopic examination of hematoxylin and eosin (H&E) stained slides has been the cornerstone of anatomic pathology diagnosis for over a century. Virtually every surgical specimen is processed, embedded in paraffin, cut, mounted on a slide, and stained in a histology laboratory. Anatomic pathology laboratory processes contain a multitude of technical and manual workflow steps which can be further complicated by being located offsite. Automated instruments, particularly tissue processors and stainers, have accelerated histology lab work. However, important manual tasks such as specimen grossing, tissue cassetting, embedding, cutting, mounting, and tissue examination under a microscope for quality control remain largely unautomated. If these processes are not closely monitored, the quality of patient care suffers.\nMost processes in a modern histology lab are captured by an integrated laboratory information system (LIS). Data visualization is a powerful tool which can illustrate workflow trends using charts, diagrams, and tables.[1][2][3] The use of data visualization on captured histology data elements from the LIS grants the ability to monitor nearly all steps of histology lab operation. Careful upstream capture of workflow steps and their corresponding data elements in the LIS can help build robust data visualizations, leading to rapid identification of processes and their improvement.[3] This improves the accuracy and timely reporting of the final pathology diagnosis, which directly affects patient care and compensation.\nOur institution, the University of Florida, serves a large, roughly 1000-bed tertiary care hospital with multiple surgical subspecialties available. Diagnoses range from mundane to extraordinarily rare. Our histology laboratory also acts as an outreach lab for a number of regional clinics and is geographically separated from our main hospital and anatomic pathology offices. Our pathology department uses Beaker (Epic Systems Corporation, Verona, WI, USA) as our LIS. Beaker, as with other LIS platforms, makes use of 1D and 2D bar code reading for patient and specimen identification. Labs can use these to track specimen movement in exquisite detail and can be leveraged to evaluate anatomic pathology lab performance. We found such tracking to be essential given our lab\u2019s geographical separation from the hospital and pathologists. \nTo our knowledge, few authors have explored data visualization and anatomic pathology workflows. Standout work from the University of Iowa[4] and Duke University[5] provided our institution with guidance during our Beaker implementation. However, those authors did not fully explore the potential of data visualization for quality improvement. Using a case-oriented format, this report demonstrates data visualization techniques, as applied to anatomic pathology workflows and may serve as a guide for others.\n\nEncountered challenges and tailored informatics solutions \nHistotechnologist status board \nIn Beaker, pending histology work is organized in a module called the \u201ccase-prep worklist.\u201d A major deficiency in this module is that it is case-based and not task-based. Every line in the case-prep worklist represents one case, which can have multiple tasks (e.g., see the incomplete H&E, MUM-1, and Her2 tasks in Figure 1A). In a complex lab, histotechnologists work on dedicated sets of tasks and do not work in a case-based manner. For example, a histotechnologist may only work on biopsy H&E slides for their shift.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. (A) The case-prep work list organizes pending histology tasks by case with a truncated list, the \u201cIncomplete\u201d column, to show the actual pending task types. Specific tasks are \u201cconfirmed\u201d on the right-hand panel. (B) Our custom \u201cstatus board\u201d is displayed in the lab similar to inpatient ward status boards. Here histology tasks are listed with their pertinent information, at-a-glance.\n\n\n\nWithin Beaker, pending tasks are presented as a single, concatenated and truncated string within a column (see the \u201cIncomplete\u201d column of Figure 1A) with no option for sorting, filtering, or expanding. Histotechnologists must click on a case in the list to view all pending tasks in a separate individual pane (see right-hand pane in Figure 1A, Steps 1 and 2). To begin work on a slide, a histotechnologist needs to mark the individual task as \u201cconfirmed\u201d (Figure 1A, Step 3). At this point, a unique label is printed for the slide, and the task disappears from the case-prep worklist (Figure 1A). This is disadvantageous since the system considers the task to be complete at this point when, in fact, the work (technical process of slide preparation) is just beginning. Also, there is minimal flexibility in changing task parameters after it is confirmed. Specific task details, such as rush priority, as well as the tasks themselves, can be \u201cburied\u201d amid other information. We found the overall histotechnologist user experience for the case-prep worklist unintuitive during initial Beaker implementation and felt the need for a task-centric worklist.\nTo address these drawbacks, we explored other workflow organization techniques implemented in Epic modules in other areas of the hospital, namely the inpatient\/emergency department \u201cstatus boards.\u201d Status boards, commonly found in nursing areas, display key information regarding patient care in an organized way, tailored to that specific area. With this template in mind, we generated a task-centric \u201cstatus board\u201d within Beaker for use in the histology laboratory. We implemented the status board using Beaker's MyReports module by leveraging tailored report settings (Figure 1B). This status board displayed pending histology lab tasks, priorities, and time-in-status among other task-specific notes that the ordering user may provide. This allowed for triage of urgent cases and helped communicate case-specific needs at-a-glance. The status board has a filter and sort features. This bolsters the user experience, visualizing all specimens that require the same task. For example, the user can filter the status board by \u201cH&E biopsy\u201d tasks and then sort by date received to ensure the oldest tasks are addressed first. While doing this, the user would also be able to note tasks flagged as \u201cRUSH,\u201d as well as specific notes from the grossing staff such as \u201ctiny biopsy, three total pieces.\u201d Lastly, the status board could adapt to changes in slide routing, such as a case being assigned to a different pathologist while in process, which was not possible under the case-prep worklist. When slide tasks are shipped out of the lab, they are automatically removed from the status board, mimicking the actual workflow. Our status board in many ways supplanted the default case-prep worklist used by Beaker.\nOur status board is also useful for intraoperative slide logistics. At our institution, all intraoperative slides accompany the tissue cassettes to histology, after specimen grossing. Ideally, all diagnostic case materials would then remain together through histology lab generation of permanent tissue sections from the cassettes. Unfortunately, intraoperative slides would be missed when staff were packing permanent sections for a given case and were not sent to the main hospital. We found that this occurred because there was no indicator to alert the staff to locate intraoperative slides at the time of case packing. An additional benefit of the status board is that it also acts as an inventory for all case materials which need to be shipped to the pathologist at a specific time. Thus, unlike any default Beaker module, we created a status board to display the intraoperative slides to be shipped along with the permanent sections.\n\nSlide volume, compensation, and resident training \nGross tissue specimens are examined by both trained professionals (pathology assistants) as well as pathologists in training (pathology residents). Certain trainee mistakes may be overlooked by mentors and staff. Routine lab volume reporting failed to identify an increase in block volumes from cases prepared by grossing-room trainees. Specifically, excess blocks were submitted for routine mammoplasty (non-cancerous) specimens by trainees. This spike in volume delayed the processing of these specimens and impeded other lab tasks on the days in which excess blocks were submitted. To evaluate the scope and severity of the problem, we developed a data dashboard. This helped to identify the trainees driving the issue. Additionally, these data were used in conjunction with compensation data to visualize the drop in case compensation with the submission of excess blocks (and by extension, slides) on these types of cases (Figure 2A). This visualization proved useful in trainee education and corrected the blocks-per-case levels.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. This composite figure shows a rendering of data generated by our SAP Business Objects dashboards. These graphs are continuously reviewed, but, as described in their relevant sections, SAP Business Objects functionality allowed for drill-down and filtering data elements to investigate and solve various histology lab problems. (A) A plot of slides per non-cancerous mammoplasty case by compensation illustrates the inverse relationship between the number of blocks submitted and the net revenue. (B) Prostate biopsy specimens from a specific client (dashed line) and overall (solid line) histology TAT. (C) Plot of hours from slide completion in histology to slide delivery to pathologist by month. This delay was ameliorated by changing courier procedure. (D) A daily surge in IHC order volume was identified immediately prior to the 10 a.m. \u201csame day\u201d cutoff. This surge, and its effects on case turnaround time, assisted in lobbying for additional instrumentation.\n\n\n\nSubspecialty-specific turnaround time investigation \nTurnaround time (TAT) is calculated from the time of specimen accessioning to the time when the final pathology report is finalized. In 2019, histology leadership was notified by Genitourinary sub-specialist pathologists that our prostate biopsy TAT was slow. This was initially misattributed to a decrease in staffing, which affected the overall histology lab TAT for September to November 2019 (see Figure 2B). The prostate subspecialty-specific drop-in TAT compliance was investigated by building a dynamic TAT reporting dashboard with the capability to filter across multiple variables (e.g., surgical specialty, specimen source, contributing clinic). Using this dashboard, we filtered the cases by prostate subspecialty and client, which revealed that the delay was not due to decreased staffing but rather an idiosyncrasy in case handling. The delayed prostate specimens exclusively belonged to a specific outpatient surgical center. The case handling error was traced to the placement of the biopsies in question in a nonstandard, overlooked location. Correction of the error led to 100 percent TAT compliance by the following month (October 2019, as shown in Figure 2B).\n\nDelayed slide delivery \nSeveral end-user pathologists reported that the histology lab was taking longer to complete outreach cases than in-house cases. Histology lab TAT reports showed no delays during the time period investigated. However, transit times for cases were not included in the histology TAT reports. To identify the source of delay, we extracted previously unevaluated timestamps from the Epic Clarity database. Timestamps are recorded in this database every time a slide is scanned, but they were not necessarily being used to evaluate lab efficiency.\nSAP BusinessObjects was used to visually interrogate the time-delta of every step a histology slide experienced, from specimen processing to individual slide delivery (e.g., as slide creation, packing out or received times).\nThis data visualization solution quickly identified the source of delay. The delay was due to slides being placed in a sorting\/packing area (in the histology lab) that was not checked by courier staff after 4 p.m. Thus, all slides produced after 4 p.m. missed the evening courier to surgical pathology despite being packed out of the histology lab in a timely manner. This step was not captured by routine histology TAT because the TAT captures only until a slide is scanned onto a packing list. Once identified, the error was quickly corrected and the average time to slide delivery normalized (see Figure 2C).\n\nSame-day immunohistochemistry order volume surge detection \nOur lab offers same-day delivery for immunohistochemistry (IHC) orders placed before 10 a.m. Histotechnologists felt that most IHC orders arrived just before the 10 a.m. deadline, limiting capacity by virtue of the number of instruments and insufficient staffing. We used Epic Beaker and SAP BusinessObjects to generate histograms of IHC order volume by time (Figure 2D). We were able to identify a surge in IHC ordering between 9:30 and 10:00 a.m., corresponding to the strain experienced by histology staff. Figure 2D shows an example of a spike in order placement between 9:30 and 10:00 a.m. Identification of order surges like this, and presentation of such information in an easily digestible format to administrators, allowed us to successfully campaign for additional IHC instruments and reorganization of human resources. Being able to generate these kinds of visualizations can help lab leadership lobby for resources from hospital administration.\n\nSpecimen packing and paperwork deviation \nAll the slides generated in the histology lab have a unique barcode, which allows precise tracking within the LIS. Slides may be sent to pathologists in other buildings in our healthcare system. Beaker LIS requires the creation of a \u201cpacking list,\u201d which is a printable inventory of all the material (slides\/blocks) being shipped from one location to another. A hardcopy of the printed packing list accompanies the specimens in transit. Items in the packing list are scanned at the destination as \u201creceived\u201d to complete the digital paper trail. Items can only be received if the packing list (at the shipping location) was marked as \u201cclosed\u201d prior to shipping.\nDifficulties arose in our lab due to \u201cun-closed\u201d packing lists. For example, specimens sent from histology lab to the surgical pathology sign out area could not be scanned as received if the packing list status is not \u201cclosed\u201d at the point of origin. This created a work stoppage at the destination, as Beaker LIS prevents \u201cclosing\u201d a packing list if the user logged in at the destination. Correction of this error at the destination involves a convoluted process of switching contexts (i.e., virtual location) within Epic Beaker, effectively tricking the LIS into thinking one is at the point of origin (histology lab) instead of the destination (surgical pathology). After marking a packing list as \u201cclosed,\u201d the user must switch the context again to receive the slides. This created needless delays in patient care and frustration on the part of the technologists, owing to the frequency of this error.\nOur solution involved the addition of a large warning to the printed hard copy of a packing list. Now when a packing list is printed while \u201cun-closed,\u201d a large warning appears at the top declaring \u201cDRAFT DO NOT USE\u201d to the specimen origin technologists. This simple addition serves as a useful reminder to \u201cclose\u201d the packing list prior to the specimen leaving the location and has greatly reduced the incidence of this error.\n\nConclusion \nLIS solutions are an integral part of any modern anatomic pathology lab. However, \u201ccanned\u201d LIS implementation modules cannot fit every laboratory\u2019s needs. Our solutions demonstrated creative use of common LIS tools (Table 1). The technical and\/or functionality framework that we demonstrated in this manuscript could be adapted by other institutions to address common problems encountered by anatomic pathology laboratories. We feel our implementations, while specific to our institution\u2019s needs, are creative and nuanced in ways that may benefit other groups. Careful and continuous evaluation, post go-live, along with creative problem solving are required to fully realize the potential of electronic data tracking, ultimately leading to meaningful insights. This includes improving workflows and data analytics. Inclusive collaboration across all stakeholders, which comprises not only laboratory leadership but also \u201cground level\u201d staff, is required to bridge workflow knowledge gaps with data analytics. Additionally, our laboratory\u2019s stakeholders became more conversant in informatics approaches through a specialized departmental pathology informatics courses.[6] Our partnership led to the identification and resolution of multiple workflow issues that could arise in any modern histology lab. It is our hope that other histology lab leaders can find utility and inspiration from our experience.\n\n\n\n\n\n\n\nTable 1. Lessons learned\n\n\nChallenge encountered\n\nCorrectional approach\n\n\nHistotechnologist status board\n\n\n Nonstandard reporting functions can be customized to display information in an efficient, workflow-oriented manner.\n This status board is used continuously by staff.\n\n\nSlide volume, compensation and resident tracking\n\n\n Resident training and policy can be developed through the utilization of histology block and slide volume data.\n Conclusions from these data are now part of resident training.\n\n\nSubspecialty-specific TAT\n\n\n Careful slicing of data with visualization can unveil TAT problems.\n Specialty-specific reports are utilized as a part of monthly QC and are part of any TAT-related investigation in the lab.\n\n\nDelayed slide delivery\n\n\n Not all steps between initial lab processing of tissue and delivery to pathologist are captured by routine TAT reports.\n Lab-to-pathologist TAT visualizations are part of any TAT-related investigation in the lab.\n\n\nSame day IHC volume surge detection\n\n\n Surges in volume related to a specific lab area can be visualized with good informatics practices.\n This report is used during meetings with administration regarding acquisition of new instruments or personnel, as well as staff scheduling.\n\n\nSpecimen packing and paperwork deviation\n\n\n Creative use of LIS features can provide an extra layer of patient safety.\n These alerts have reduced the incidence of the type of error described to near zero.\n\n\n\nAcknowledgements \nWe acknowledge UF Health Medical Labs laboratory members Chelsea Nagle, Elaine Dooley, Adrian Ferguson, and Robin Hoskins for helpful discussion about histology workflow improvement efforts. We thank Paul Sharpe, Tyler Gennaro, and David Dekay, from the UF Health enterprise IT services for their dedication in implementing these new features.\n\nAuthor contributions \nAll authors participated in the conceptualization and design of the histology informatics protocol and workflows described in this manuscript. V.C., R.S., S.G., L.W., P.C., and S.C. lead the informatics implementation efforts. V.C., R.S., N.Q., S.F., and S.C. took the lead in writing the manuscript with all authors input. All authors provided critical feedback and approved the final version.\n\nFunding \nNo funding to disclose.\n\nConflict of interest \nThe authors declare that they have no competing interests.\n\nReferences \n\n\n\u2191 \"What is Data Visualization?\". Klipfolio. https:\/\/www.klipfolio.com\/resources\/articles\/what-is-data-visualization . Retrieved 19 November 2020 .   \n\n\u2191 Few, S. (18 September 2004). \"Eenie, Meenie, Minie, Moe: Selecting the Right Graph for Your Message\" (PDF). Perceptual Edge. http:\/\/www.perceptualedge.com\/articles\/ie\/the_right_graph.pdf . Retrieved 19 November 2020 .   \n\n\u2191 3.0 3.1 Halwani, F.; Li, W.C.; Banerjee, D. et al. (2016). \"A real-time dashboard for managing pathology processes\". Journal of Pathology Informatics 7: 24. doi:10.4103\/2153-3539.181768. PMC PMC4872478. PMID 27217974. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4872478 .   \n\n\u2191 Blau, J.L.; Wilford, J.D.; Dane, S.K. et al. (2017). \"Implementation of Epic Beaker Anatomic Pathology at an Academic Medical Center\". Journal of Pathology Informatics 8: 47. doi:10.4103\/jpi.jpi_31_17. PMC PMC5760958. PMID 29387505. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5760958 .   \n\n\u2191 Albert, C. (2019). \"Ups and downs of bringing in Beaker AP LIS\". CAP Today (August 2019). https:\/\/www.captodayonline.com\/ups-and-downs-of-bringing-in-beaker-ap-lis\/ .   \n\n\u2191 Maness, H.T.; Behar-Horenstein, L.S.; Clare-Salzler, M. et al. (2020). \"Informatics Training for Pathology Practice and Research in the Digital Era\". Academic Pathology 7: 2374289520911179. doi:10.1177\/2374289520911179. PMC PMC7119238. PMID 32284963. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7119238 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation and updates to spelling and grammar. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab\">https:\/\/www.limswiki.org\/index.php\/Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on pathology informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 3 May 2021, at 20:01.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 353 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","25073682c23be675cc4cd14be481b85b_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Informatics-driven_quality_improvement_in_the_modern_histology_lab skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Informatics-driven quality improvement in the modern histology lab<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">Laboratory information systems<\/a> (LISs) and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_visualization\" title=\"Data visualization\" class=\"wiki-link\" data-key=\"4a3b86cba74bc7bb7471aa3fc2fcccc3\">data visualization<\/a> techniques have untapped potential in <a href=\"https:\/\/www.limswiki.org\/index.php\/Anatomical_pathology\" title=\"Anatomical pathology\" class=\"wiki-link\" data-key=\"5668db6faf37e8c1432a1d7953f30cb7\">anatomic pathology<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratories<\/a>. The pre-built functionalities of an LIS do not address all the needs of a modern <a href=\"https:\/\/www.limswiki.org\/index.php\/Histology\" title=\"Histology\" class=\"wiki-link\" data-key=\"3525f62ddae222c7cb9ba0159ebbea19\">histology<\/a> laboratory. For instance, \u201cgo live\u201d is not the end of LIS customization, only the beginning. After closely evaluating various histology lab , we implemented several custom <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">data analytics<\/a> dashboards and additional LIS functionalities to monitor and address weaknesses. Herein, we present our experience with LIS and data-tracking solutions that improved trainee education, slide logistics, staffing and instrumentation lobbying, and task tracking. The latter was addressed through the creation of a novel \u201cstatus board\u201d akin to those seen in inpatient wards. These use-cases can benefit other histology laboratories.\n<\/p><p><b>Lay summary<\/b>: <a href=\"https:\/\/www.limswiki.org\/index.php\/Histopathology\" title=\"Histopathology\" class=\"wiki-link\" data-key=\"1250dd919ba7b6079a53d9b17e5bc666\">Histopathology<\/a> is the branch of medicine that involves the gross and microscopic examination of <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">sampled<\/a> tissue to help reach a diagnosis. Histology lab workflows incorporate a multitude of automated and manual steps that require close monitoring. Data visualization is a powerful tool that can illustrate workflow trends using a graphic representation of data. This manuscript describes data visualization and data analysis techniques leveraged to address frequent pitfalls in histology laboratory workflows. This led to multiple improvements in patient safety and quality of care. Our use-case solutions, presented here, can be adapted by other histology laboratories to overcome common laboratory challenges.\n<\/p><p><b>Keywords<\/b>: informatics, laboratory information system, quality improvement, pathology, workflow\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>The microscopic examination of hematoxylin and eosin (H&E) stained slides has been the cornerstone of <a href=\"https:\/\/www.limswiki.org\/index.php\/Anatomical_pathology\" title=\"Anatomical pathology\" class=\"wiki-link\" data-key=\"5668db6faf37e8c1432a1d7953f30cb7\">anatomic pathology<\/a> diagnosis for over a century. Virtually every surgical specimen is processed, embedded in paraffin, cut, mounted on a slide, and stained in a <a href=\"https:\/\/www.limswiki.org\/index.php\/Histology\" title=\"Histology\" class=\"wiki-link\" data-key=\"3525f62ddae222c7cb9ba0159ebbea19\">histology<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a>. Anatomic pathology laboratory processes contain a multitude of technical and manual <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a> steps which can be further complicated by being located offsite. Automated instruments, particularly tissue processors and stainers, have accelerated histology lab work. However, important manual tasks such as specimen grossing, tissue cassetting, embedding, cutting, mounting, and tissue examination under a microscope for <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> remain largely unautomated. If these processes are not closely monitored, the quality of patient care suffers.\n<\/p><p>Most processes in a modern histology lab are captured by an integrated <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS). <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_visualization\" title=\"Data visualization\" class=\"wiki-link\" data-key=\"4a3b86cba74bc7bb7471aa3fc2fcccc3\">Data visualization<\/a> is a powerful tool which can illustrate workflow trends using charts, diagrams, and tables.<sup id=\"rdp-ebb-cite_ref-KFWhatIs_1-0\" class=\"reference\"><a href=\"#cite_note-KFWhatIs-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FewEenie04_2-0\" class=\"reference\"><a href=\"#cite_note-FewEenie04-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HalwaniAReal16_3-0\" class=\"reference\"><a href=\"#cite_note-HalwaniAReal16-3\">[3]<\/a><\/sup> The use of data visualization on captured histology data elements from the LIS grants the ability to monitor nearly all steps of histology lab operation. Careful upstream capture of workflow steps and their corresponding data elements in the LIS can help build robust data visualizations, leading to rapid identification of processes and their improvement.<sup id=\"rdp-ebb-cite_ref-HalwaniAReal16_3-1\" class=\"reference\"><a href=\"#cite_note-HalwaniAReal16-3\">[3]<\/a><\/sup> This improves the accuracy and timely reporting of the final pathology diagnosis, which directly affects patient care and compensation.\n<\/p><p>Our institution, the University of Florida, serves a large, roughly 1000-bed tertiary care <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> with multiple surgical subspecialties available. Diagnoses range from mundane to extraordinarily rare. Our histology laboratory also acts as an outreach lab for a number of regional clinics and is geographically separated from our main hospital and anatomic pathology offices. Our pathology department uses Beaker (Epic Systems Corporation, Verona, WI, USA) as our LIS. Beaker, as with other LIS platforms, makes use of 1D and 2D bar code reading for patient and <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">specimen<\/a> identification. Labs can use these to track specimen movement in exquisite detail and can be leveraged to evaluate anatomic pathology lab performance. We found such tracking to be essential given our lab\u2019s geographical separation from the hospital and pathologists. \n<\/p><p>To our knowledge, few authors have explored data visualization and anatomic pathology workflows. Standout work from the University of Iowa<sup id=\"rdp-ebb-cite_ref-BlauImplem17_4-0\" class=\"reference\"><a href=\"#cite_note-BlauImplem17-4\">[4]<\/a><\/sup> and Duke University<sup id=\"rdp-ebb-cite_ref-AlbertUps19_5-0\" class=\"reference\"><a href=\"#cite_note-AlbertUps19-5\">[5]<\/a><\/sup> provided our institution with guidance during our Beaker implementation. However, those authors did not fully explore the potential of data visualization for quality improvement. Using a case-oriented format, this report demonstrates data visualization techniques, as applied to anatomic pathology workflows and may serve as a guide for others.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Encountered_challenges_and_tailored_informatics_solutions\">Encountered challenges and tailored informatics solutions<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Histotechnologist_status_board\">Histotechnologist status board<\/span><\/h3>\n<p>In Beaker, pending histology work is organized in a module called the \u201ccase-prep worklist.\u201d A major deficiency in this module is that it is case-based and not task-based. Every line in the case-prep worklist represents one case, which can have multiple tasks (e.g., see the incomplete H&E, MUM-1, and Her2 tasks in Figure 1A). In a complex lab, histotechnologists work on dedicated sets of tasks and do not work in a case-based manner. For example, a histotechnologist may only work on biopsy H&E slides for their shift.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Seifert_JAMIAOpen20_3.png\" class=\"image wiki-link\" data-key=\"60df897494abb2d4cb1c9b7914997b54\"><img alt=\"Fig1 Seifert JAMIAOpen20 3.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/5e\/Fig1_Seifert_JAMIAOpen20_3.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> (<b>A<\/b>) The case-prep work list organizes pending histology tasks by case with a truncated list, the \u201cIncomplete\u201d column, to show the actual pending task types. Specific tasks are \u201cconfirmed\u201d on the right-hand panel. (<b>B<\/b>) Our custom \u201cstatus board\u201d is displayed in the lab similar to inpatient ward status boards. Here histology tasks are listed with their pertinent information, at-a-glance.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Within Beaker, pending tasks are presented as a single, concatenated and truncated string within a column (see the \u201cIncomplete\u201d column of Figure 1A) with no option for sorting, filtering, or expanding. Histotechnologists must click on a case in the list to view all pending tasks in a separate individual pane (see right-hand pane in Figure 1A, Steps 1 and 2). To begin work on a slide, a histotechnologist needs to mark the individual task as \u201cconfirmed\u201d (Figure 1A, Step 3). At this point, a unique label is printed for the slide, and the task disappears from the case-prep worklist (Figure 1A). This is disadvantageous since the system considers the task to be complete at this point when, in fact, the work (technical process of slide preparation) is just beginning. Also, there is minimal flexibility in changing task parameters after it is confirmed. Specific task details, such as rush priority, as well as the tasks themselves, can be \u201cburied\u201d amid other <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>. We found the overall histotechnologist user experience for the case-prep worklist unintuitive during initial Beaker implementation and felt the need for a task-centric worklist.\n<\/p><p>To address these drawbacks, we explored other workflow organization techniques implemented in Epic modules in other areas of the hospital, namely the inpatient\/emergency department \u201cstatus boards.\u201d Status boards, commonly found in nursing areas, display key information regarding patient care in an organized way, tailored to that specific area. With this template in mind, we generated a task-centric \u201cstatus board\u201d within Beaker for use in the histology laboratory. We implemented the status board using Beaker's MyReports module by leveraging tailored report settings (Figure 1B). This status board displayed pending histology lab tasks, priorities, and time-in-status among other task-specific notes that the ordering user may provide. This allowed for triage of urgent cases and helped communicate case-specific needs at-a-glance. The status board has a filter and sort features. This bolsters the user experience, visualizing all specimens that require the same task. For example, the user can filter the status board by \u201cH&E biopsy\u201d tasks and then sort by date received to ensure the oldest tasks are addressed first. While doing this, the user would also be able to note tasks flagged as \u201cRUSH,\u201d as well as specific notes from the grossing staff such as \u201ctiny biopsy, three total pieces.\u201d Lastly, the status board could adapt to changes in slide routing, such as a case being assigned to a different pathologist while in process, which was not possible under the case-prep worklist. When slide tasks are shipped out of the lab, they are automatically removed from the status board, mimicking the actual workflow. Our status board in many ways supplanted the default case-prep worklist used by Beaker.\n<\/p><p>Our status board is also useful for intraoperative slide logistics. At our institution, all intraoperative slides accompany the tissue cassettes to histology, after specimen grossing. Ideally, all diagnostic case materials would then remain together through histology lab generation of permanent tissue sections from the cassettes. Unfortunately, intraoperative slides would be missed when staff were packing permanent sections for a given case and were not sent to the main hospital. We found that this occurred because there was no indicator to alert the staff to locate intraoperative slides at the time of case packing. An additional benefit of the status board is that it also acts as an inventory for all case materials which need to be shipped to the pathologist at a specific time. Thus, unlike any default Beaker module, we created a status board to display the intraoperative slides to be shipped along with the permanent sections.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Slide_volume.2C_compensation.2C_and_resident_training\">Slide volume, compensation, and resident training<\/span><\/h3>\n<p>Gross tissue specimens are examined by both trained professionals (pathology assistants) as well as pathologists in training (pathology residents). Certain trainee mistakes may be overlooked by mentors and staff. Routine lab volume reporting failed to identify an increase in block volumes from cases prepared by grossing-room trainees. Specifically, excess blocks were submitted for routine mammoplasty (non-cancerous) specimens by trainees. This spike in volume delayed the processing of these specimens and impeded other lab tasks on the days in which excess blocks were submitted. To evaluate the scope and severity of the problem, we developed a data dashboard. This helped to identify the trainees driving the issue. Additionally, these data were used in conjunction with compensation data to visualize the drop in case compensation with the submission of excess blocks (and by extension, slides) on these types of cases (Figure 2A). This visualization proved useful in trainee education and corrected the blocks-per-case levels.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Seifert_JAMIAOpen20_3.png\" class=\"image wiki-link\" data-key=\"d581732c646a83b22ca2c221a819ac27\"><img alt=\"Fig2 Seifert JAMIAOpen20 3.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/23\/Fig2_Seifert_JAMIAOpen20_3.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> This composite figure shows a rendering of data generated by our SAP Business Objects dashboards. These graphs are continuously reviewed, but, as described in their relevant sections, SAP Business Objects functionality allowed for drill-down and filtering data elements to investigate and solve various histology lab problems. (<b>A<\/b>) A plot of slides per non-cancerous mammoplasty case by compensation illustrates the inverse relationship between the number of blocks submitted and the net revenue. (<b>B<\/b>) Prostate biopsy specimens from a specific client (dashed line) and overall (solid line) histology TAT. (<b>C<\/b>) Plot of hours from slide completion in histology to slide delivery to pathologist by month. This delay was ameliorated by changing courier procedure. (<b>D<\/b>) A daily surge in IHC order volume was identified immediately prior to the 10 a.m. \u201csame day\u201d cutoff. This surge, and its effects on case turnaround time, assisted in lobbying for additional instrumentation.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Subspecialty-specific_turnaround_time_investigation\">Subspecialty-specific turnaround time investigation<\/span><\/h3>\n<p>Turnaround time (TAT) is calculated from the time of specimen accessioning to the time when the final pathology report is finalized. In 2019, histology leadership was notified by Genitourinary sub-specialist pathologists that our prostate biopsy TAT was slow. This was initially misattributed to a decrease in staffing, which affected the overall histology lab TAT for September to November 2019 (see Figure 2B). The prostate subspecialty-specific drop-in TAT compliance was investigated by building a dynamic TAT reporting dashboard with the capability to filter across multiple variables (e.g., surgical specialty, specimen source, contributing clinic). Using this dashboard, we filtered the cases by prostate subspecialty and client, which revealed that the delay was not due to decreased staffing but rather an idiosyncrasy in case handling. The delayed prostate specimens exclusively belonged to a specific outpatient surgical center. The case handling error was traced to the placement of the biopsies in question in a nonstandard, overlooked location. Correction of the error led to 100 percent TAT compliance by the following month (October 2019, as shown in Figure 2B).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Delayed_slide_delivery\">Delayed slide delivery<\/span><\/h3>\n<p>Several end-user pathologists reported that the histology lab was taking longer to complete outreach cases than in-house cases. Histology lab TAT reports showed no delays during the time period investigated. However, transit times for cases were not included in the histology TAT reports. To identify the source of delay, we extracted previously unevaluated timestamps from the Epic Clarity database. Timestamps are recorded in this database every time a slide is scanned, but they were not necessarily being used to evaluate lab efficiency.\n<\/p><p>SAP BusinessObjects was used to visually interrogate the time-delta of every step a histology slide experienced, from specimen processing to individual slide delivery (e.g., as slide creation, packing out or received times).\n<\/p><p>This data visualization solution quickly identified the source of delay. The delay was due to slides being placed in a sorting\/packing area (in the histology lab) that was not checked by courier staff after 4 p.m. Thus, all slides produced after 4 p.m. missed the evening courier to surgical pathology despite being packed out of the histology lab in a timely manner. This step was not captured by routine histology TAT because the TAT captures only until a slide is scanned onto a packing list. Once identified, the error was quickly corrected and the average time to slide delivery normalized (see Figure 2C).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Same-day_immunohistochemistry_order_volume_surge_detection\">Same-day immunohistochemistry order volume surge detection<\/span><\/h3>\n<p>Our lab offers same-day delivery for immunohistochemistry (IHC) orders placed before 10 a.m. Histotechnologists felt that most IHC orders arrived just before the 10 a.m. deadline, limiting capacity by virtue of the number of instruments and insufficient staffing. We used Epic Beaker and SAP BusinessObjects to generate histograms of IHC order volume by time (Figure 2D). We were able to identify a surge in IHC ordering between 9:30 and 10:00 a.m., corresponding to the strain experienced by histology staff. Figure 2D shows an example of a spike in order placement between 9:30 and 10:00 a.m. Identification of order surges like this, and presentation of such information in an easily digestible format to administrators, allowed us to successfully campaign for additional IHC instruments and reorganization of human resources. Being able to generate these kinds of visualizations can help lab leadership lobby for resources from hospital administration.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Specimen_packing_and_paperwork_deviation\">Specimen packing and paperwork deviation<\/span><\/h3>\n<p>All the slides generated in the histology lab have a unique <a href=\"https:\/\/www.limswiki.org\/index.php\/Barcode\" title=\"Barcode\" class=\"wiki-link\" data-key=\"e0952b5b262392be0995237aec36d355\">barcode<\/a>, which allows precise tracking within the LIS. Slides may be sent to pathologists in other buildings in our healthcare system. Beaker LIS requires the creation of a \u201cpacking list,\u201d which is a printable inventory of all the material (slides\/blocks) being shipped from one location to another. A hardcopy of the printed packing list accompanies the specimens in transit. Items in the packing list are scanned at the destination as \u201creceived\u201d to complete the digital paper trail. Items can only be received if the packing list (at the shipping location) was marked as \u201cclosed\u201d prior to shipping.\n<\/p><p>Difficulties arose in our lab due to \u201cun-closed\u201d packing lists. For example, specimens sent from histology lab to the surgical pathology sign out area could not be scanned as received if the packing list status is not \u201cclosed\u201d at the point of origin. This created a work stoppage at the destination, as Beaker LIS prevents \u201cclosing\u201d a packing list if the user logged in at the destination. Correction of this error at the destination involves a convoluted process of switching contexts (i.e., virtual location) within Epic Beaker, effectively tricking the LIS into thinking one is at the point of origin (histology lab) instead of the destination (surgical pathology). After marking a packing list as \u201cclosed,\u201d the user must switch the context again to receive the slides. This created needless delays in patient care and frustration on the part of the technologists, owing to the frequency of this error.\n<\/p><p>Our solution involved the addition of a large warning to the printed hard copy of a packing list. Now when a packing list is printed while \u201cun-closed,\u201d a large warning appears at the top declaring \u201cDRAFT DO NOT USE\u201d to the specimen origin technologists. This simple addition serves as a useful reminder to \u201cclose\u201d the packing list prior to the specimen leaving the location and has greatly reduced the incidence of this error.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>LIS solutions are an integral part of any modern anatomic pathology lab. However, \u201ccanned\u201d LIS implementation modules cannot fit every laboratory\u2019s needs. Our solutions demonstrated creative use of common LIS tools (Table 1). The technical and\/or functionality framework that we demonstrated in this manuscript could be adapted by other institutions to address common problems encountered by anatomic pathology laboratories. We feel our implementations, while specific to our institution\u2019s needs, are creative and nuanced in ways that may benefit other groups. Careful and continuous evaluation, post go-live, along with creative problem solving are required to fully realize the potential of electronic data tracking, ultimately leading to meaningful insights. This includes improving workflows and data analytics. Inclusive collaboration across all stakeholders, which comprises not only laboratory leadership but also \u201cground level\u201d staff, is required to bridge workflow knowledge gaps with data analytics. Additionally, our laboratory\u2019s stakeholders became more conversant in informatics approaches through a specialized departmental pathology informatics courses.<sup id=\"rdp-ebb-cite_ref-ManessInform20_6-0\" class=\"reference\"><a href=\"#cite_note-ManessInform20-6\">[6]<\/a><\/sup> Our partnership led to the identification and resolution of multiple workflow issues that could arise in any modern histology lab. It is our hope that other histology lab leaders can find utility and inspiration from our experience.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> Lessons learned\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Challenge encountered\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Correctional approach\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Histotechnologist status board\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<ul><li> Nonstandard reporting functions can be customized to display information in an efficient, workflow-oriented manner.<\/li>\n<li> This status board is used continuously by staff.<\/li><\/ul>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Slide volume, compensation and resident tracking\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<ul><li> Resident training and policy can be developed through the utilization of histology block and slide volume data.<\/li>\n<li> Conclusions from these data are now part of resident training.<\/li><\/ul>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Subspecialty-specific TAT\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<ul><li> Careful slicing of data with visualization can unveil TAT problems.<\/li>\n<li> Specialty-specific reports are utilized as a part of monthly QC and are part of any TAT-related investigation in the lab.<\/li><\/ul>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Delayed slide delivery\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<ul><li> Not all steps between initial lab processing of tissue and delivery to pathologist are captured by routine TAT reports.<\/li>\n<li> Lab-to-pathologist TAT visualizations are part of any TAT-related investigation in the lab.<\/li><\/ul>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Same day IHC volume surge detection\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<ul><li> Surges in volume related to a specific lab area can be visualized with good informatics practices.<\/li>\n<li> This report is used during meetings with administration regarding acquisition of new instruments or personnel, as well as staff scheduling.<\/li><\/ul>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Specimen packing and paperwork deviation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<ul><li> Creative use of LIS features can provide an extra layer of patient safety.<\/li>\n<li> These alerts have reduced the incidence of the type of error described to near zero.<\/li><\/ul>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We acknowledge UF Health Medical Labs laboratory members Chelsea Nagle, Elaine Dooley, Adrian Ferguson, and Robin Hoskins for helpful discussion about histology workflow improvement efforts. We thank Paul Sharpe, Tyler Gennaro, and David Dekay, from the UF Health enterprise IT services for their dedication in implementing these new features.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>All authors participated in the conceptualization and design of the histology informatics protocol and workflows described in this manuscript. V.C., R.S., S.G., L.W., P.C., and S.C. lead the informatics implementation efforts. V.C., R.S., N.Q., S.F., and S.C. took the lead in writing the manuscript with all authors input. All authors provided critical feedback and approved the final version.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>No funding to disclose.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-KFWhatIs-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KFWhatIs_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.klipfolio.com\/resources\/articles\/what-is-data-visualization\" target=\"_blank\">\"What is Data Visualization?\"<\/a>. <i>Klipfolio<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.klipfolio.com\/resources\/articles\/what-is-data-visualization\" target=\"_blank\">https:\/\/www.klipfolio.com\/resources\/articles\/what-is-data-visualization<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 November 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+is+Data+Visualization%3F&rft.atitle=Klipfolio&rft_id=https%3A%2F%2Fwww.klipfolio.com%2Fresources%2Farticles%2Fwhat-is-data-visualization&rfr_id=info:sid\/en.wikipedia.org:Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FewEenie04-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FewEenie04_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Few, S. (18 September 2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.perceptualedge.com\/articles\/ie\/the_right_graph.pdf\" target=\"_blank\">\"Eenie, Meenie, Minie, Moe: Selecting the Right Graph for Your Message\"<\/a> (PDF). <i>Perceptual Edge<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.perceptualedge.com\/articles\/ie\/the_right_graph.pdf\" target=\"_blank\">http:\/\/www.perceptualedge.com\/articles\/ie\/the_right_graph.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 November 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Eenie%2C+Meenie%2C+Minie%2C+Moe%3A+Selecting+the+Right+Graph+for+Your+Message&rft.atitle=Perceptual+Edge&rft.aulast=Few%2C+S.&rft.au=Few%2C+S.&rft.date=18+September+2004&rft_id=http%3A%2F%2Fwww.perceptualedge.com%2Farticles%2Fie%2Fthe_right_graph.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HalwaniAReal16-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HalwaniAReal16_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-HalwaniAReal16_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Halwani, F.; Li, W.C.; Banerjee, D. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4872478\" target=\"_blank\">\"A real-time dashboard for managing pathology processes\"<\/a>. <i>Journal of Pathology Informatics<\/i> <b>7<\/b>: 24. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2F2153-3539.181768\" target=\"_blank\">10.4103\/2153-3539.181768<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4872478\/\" target=\"_blank\">PMC4872478<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27217974\" target=\"_blank\">27217974<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4872478\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4872478<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+real-time+dashboard+for+managing+pathology+processes&rft.jtitle=Journal+of+Pathology+Informatics&rft.aulast=Halwani%2C+F.%3B+Li%2C+W.C.%3B+Banerjee%2C+D.+et+al.&rft.au=Halwani%2C+F.%3B+Li%2C+W.C.%3B+Banerjee%2C+D.+et+al.&rft.date=2016&rft.volume=7&rft.pages=24&rft_id=info:doi\/10.4103%2F2153-3539.181768&rft_id=info:pmc\/PMC4872478&rft_id=info:pmid\/27217974&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4872478&rfr_id=info:sid\/en.wikipedia.org:Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlauImplem17-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlauImplem17_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blau, J.L.; Wilford, J.D.; Dane, S.K. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5760958\" target=\"_blank\">\"Implementation of Epic Beaker Anatomic Pathology at an Academic Medical Center\"<\/a>. <i>Journal of Pathology Informatics<\/i> <b>8<\/b>: 47. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2Fjpi.jpi_31_17\" target=\"_blank\">10.4103\/jpi.jpi_31_17<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5760958\/\" target=\"_blank\">PMC5760958<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29387505\" target=\"_blank\">29387505<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5760958\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5760958<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementation+of+Epic+Beaker+Anatomic+Pathology+at+an+Academic+Medical+Center&rft.jtitle=Journal+of+Pathology+Informatics&rft.aulast=Blau%2C+J.L.%3B+Wilford%2C+J.D.%3B+Dane%2C+S.K.+et+al.&rft.au=Blau%2C+J.L.%3B+Wilford%2C+J.D.%3B+Dane%2C+S.K.+et+al.&rft.date=2017&rft.volume=8&rft.pages=47&rft_id=info:doi\/10.4103%2Fjpi.jpi_31_17&rft_id=info:pmc\/PMC5760958&rft_id=info:pmid\/29387505&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5760958&rfr_id=info:sid\/en.wikipedia.org:Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbertUps19-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlbertUps19_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Albert, C. (2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.captodayonline.com\/ups-and-downs-of-bringing-in-beaker-ap-lis\/\" target=\"_blank\">\"Ups and downs of bringing in Beaker AP LIS\"<\/a>. <i>CAP Today<\/i> (August 2019)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.captodayonline.com\/ups-and-downs-of-bringing-in-beaker-ap-lis\/\" target=\"_blank\">https:\/\/www.captodayonline.com\/ups-and-downs-of-bringing-in-beaker-ap-lis\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ups+and+downs+of+bringing+in+Beaker+AP+LIS&rft.jtitle=CAP+Today&rft.aulast=Albert%2C+C.&rft.au=Albert%2C+C.&rft.date=2019&rft.issue=August+2019&rft_id=https%3A%2F%2Fwww.captodayonline.com%2Fups-and-downs-of-bringing-in-beaker-ap-lis%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ManessInform20-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ManessInform20_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maness, H.T.; Behar-Horenstein, L.S.; Clare-Salzler, M. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7119238\" target=\"_blank\">\"Informatics Training for Pathology Practice and Research in the Digital Era\"<\/a>. <i>Academic Pathology<\/i> <b>7<\/b>: 2374289520911179. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F2374289520911179\" target=\"_blank\">10.1177\/2374289520911179<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7119238\/\" target=\"_blank\">PMC7119238<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32284963\" target=\"_blank\">32284963<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7119238\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7119238<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Informatics+Training+for+Pathology+Practice+and+Research+in+the+Digital+Era&rft.jtitle=Academic+Pathology&rft.aulast=Maness%2C+H.T.%3B+Behar-Horenstein%2C+L.S.%3B+Clare-Salzler%2C+M.+et+al.&rft.au=Maness%2C+H.T.%3B+Behar-Horenstein%2C+L.S.%3B+Clare-Salzler%2C+M.+et+al.&rft.date=2020&rft.volume=7&rft.pages=2374289520911179&rft_id=info:doi\/10.1177%2F2374289520911179&rft_id=info:pmc\/PMC7119238&rft_id=info:pmid\/32284963&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7119238&rfr_id=info:sid\/en.wikipedia.org:Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation and updates to spelling and grammar. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150627\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.230 seconds\nReal time usage: 0.807 seconds\nPreprocessor visited node count: 6506\/1000000\nPreprocessor generated node count: 29404\/1000000\nPost\u2010expand include size: 49977\/2097152 bytes\nTemplate argument size: 16887\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 151.105 1 - -total\n 67.52% 102.023 1 - Template:Reflist\n 53.40% 80.689 6 - Template:Citation\/core\n 39.79% 60.130 4 - Template:Cite_journal\n 25.66% 38.772 1 - Template:Infobox_journal_article\n 24.56% 37.109 1 - Template:Infobox\n 18.04% 27.260 2 - Template:Cite_web\n 16.29% 24.615 80 - Template:Infobox\/row\n 6.83% 10.322 9 - Template:Citation\/identifier\n 5.21% 7.874 8 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12509-0!*!0!!en!5!* and timestamp 20210705150626 and revision id 42449\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab\">https:\/\/www.limswiki.org\/index.php\/Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","25073682c23be675cc4cd14be481b85b_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/5e\/Fig1_Seifert_JAMIAOpen20_3.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/23\/Fig2_Seifert_JAMIAOpen20_3.png"],"25073682c23be675cc4cd14be481b85b_timestamp":1625497586,"d8b467af534a70312a21f63b61be26cd_type":"article","d8b467af534a70312a21f63b61be26cd_title":"The Application of Informatics to Scientific Work: Laboratory Informatics for Newbies (Liscouski 2021)","d8b467af534a70312a21f63b61be26cd_url":"https:\/\/www.limswiki.org\/index.php\/LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies","d8b467af534a70312a21f63b61be26cd_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tLII:The Application of Informatics to Scientific Work: Laboratory Informatics for Newbies\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tTitle: The Application of Informatics to Scientific Work: Laboratory Informatics for Newbies\nAuthor for citation: Joe Liscouski, with editorial modifications by Shawn Douglas\nLicense for content: Creative Commons Attribution-ShareAlike 4.0 International\nPublication date: April 2021\n\nContents\n\n1 Introduction \n\n1.1 Intended audience \n\n\n2 Types of scientific and laboratory work \n\n2.1 Basic and applied research \n\n2.1.1 The research process \n\n\n\n\n3 Testing laboratories \n4 The laboratory notebook \n5 Using laboratory records \n6 Bringing informatics into the lab \n\n6.1 Backup strategy \n6.2 Witness review or sign-off \n6.3 Instrument-computer integration \n6.4 Expanding the research team \n\n\n7 Meeting the needs of the testing laboratory \n8 Other laboratory informatics systems \n\n8.1 Scientific data management system (SDMS) \n8.2 Laboratory execution system (LES) \n8.3 Instrument data system (IDS) \n\n\n9 Planning for laboratory informatics \n\n9.1 Education \n9.2 Planning \n9.3 Why projects fail \n\n\n10 Closing \n11 Footnotes \n12 About the author \n13 References \n\n\n\nIntroduction \nThe purpose of this piece is to introduce people who are not intimately familiar with laboratory work to the basics of laboratory operations and the role that informatics can play in assisting scientists, engineers, and technicians in their efforts. The concepts are important because they provide a functional foundation for understanding lab work and how that work is done in the early part of the twenty-first century (things will change, just wait for it).\n\nIntended audience \nThis material is intended for anyone who is interested in seeing how modern informatics tools can help those doing scientific work. It will provide an orientation to scientific and laboratory work, as well as the systems that have been developed to make that work more productive. It\u2019s for people coming out of school who have carried out lab experiments but not corporate research projects, for those who need to understand how testing labs work, and for IT professionals who may be faced with supporting computing systems in lab environments. It\u2019s also for those who may be tasked with managing projects to choose, install, and make informatics tools useful.\nFigure 1 shows the elements we\u2019ll be discussing in this piece. The treatment of the technical material will be on the lighter side, leaving in-depth subject matter to other works. Instrument data systems will be covered lightly, as any serious discussion becomes lengthy and discipline-specific very quickly; additionally, that material has been covered in other works.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Elements we\u2019ll be covering\n\n\n\nTypes of scientific and laboratory work \nScience is about seeking truthful answers to questions. Sometimes those questions are open-ended without any idea where they will lead you in answering them (e.g. \u201cWhy does water ice float?\u201d). Others are very specific, concerning material composition or properties (e.g., \u201cHow much lead is in this drinking water?\u201d, \u201cHow much does a butterfly weigh?\u201d). Still others may take some effort before you determine the best approach to working on them. The approach someone uses to address these questions depends on the nature of the question; some are destined for research, while others are addressed using specific test methods.\nThere are two types of research: basic and applied. Both can include field work, observations, experiments, models (mathematical, computer, and simulation), etc. Applied research is also done in testing or service laboratories, as with, for example, the development of new methods of analysis.\n\nBasic and applied research \nBasic research is open-ended, as you are looking into something without any idea of where the work will lead. It is often funded by grants through universities or government institutions; continued support depends on the perceived value of the research. Projects can range in size from the work of a single individual to a small team to large-scale groups studying astronomy, high-energy physics, engineering, the life sciences, or a number of fields.\nApplied research, on the other hand, is directed toward a goal. That goal could be a cure for a disease, the development of a COVID-19 vaccine, or work towards artificial intelligence (AI). As with basic research, until some early goals have been reached the work may begin with a single individual or a small team, and then the project scales up. The effort may be broken down into a set of more narrowly focused efforts, whose results will be combined as the development proceeds. Since applied research is goal-directed, funding will depend upon who benefits from those goals being met. Projects of national interest, including security, may be wholly or partially funded by the government. Projects with a commercial interest tend to be funded by corporate interests, including individual companies in their own laboratories or through contract research organizations with expertise useful to the program. Where there is interest from a number of corporate and\/or government groups, consortiums may form to distribute the cost and share in the results.\nBoth basic and applied research can be found in government institutions (including military groups, research and development agencies like the Defense Advanced Research Project Agency [DARPA], and task-specific institutions such as the National Institutes of Health [NIH]), public and private non-profit groups, corporations, consortia, and contract research organizations.\n\nThe research process \nThe research process begins with a question. Any question will do, including \u201cwhy is the sky blue?\u201d We\u2019ll bypass Greek mythology[a] by asking more questions and planning how to proceed to answer them. For example, \u201cIs the sky always blue?\u201d, \u201cWhen is\/isn\u2019t it?\u201d, and \u201cWhat other colors can it be?\u201d Once the process begins, it can include a number of steps, the choice and direction depending upon the nature of the research and the mindset of the researcher:\n\n Observations: This includes basic note-taking with support material (text, photos, drawings, charts, and scanned material). Research (e.g., as with basic astronomy, field biology) can be as simple as looking something up on Google or as complex as understanding how a virus works. Research is about asking questions and looking for answers, which often leads to more questions. It\u2019s a little like my granddaughter who always asks \u201cwhy?\u201d no matter how well I answer the previous question (or at least how well I think I did).\n Enhanced observations: This includes interacting with items under observation, as well as non-directed interactions, preliminary data gathering, and behavioral analysis.\n Experiments and information gathering: This includes organized experiments that are planned, directed, and purpose-driven, as well as data and information gathering.\n Team building: This includes the creation of teams or networks of people working on the same or similar projects.\n Analytics and reporting: This includes data and information analysis, data modeling (e.g., mathematical, computer algorithm, and simulation), information synthesis, and knowledge creation.\n Technology acquisition: This includes gaining access to public, commercial, remote, and other types of databases to assist the research.\nPinning down a \u201ctypical\u201d approach to research isn\u2019t possible because the routes people follow are as individual as the researchers and their area of work are. However, this is generally not the case with testing labs.\n\nTesting laboratories \nIn addition to research labs, there are also testing or \"service\" laboratories. Service labs carry out specific test routines on samples and specimens; you may be familiar with them as quality control labs, clinical labs, toxicology labs, forensic science labs, etc. They are called service labs because they support other organizations, including research organizations, and they have similar modes of operation and work organization, running different tests depending on their area of specialization.\nContract testing labs are another flavor of service laboratories, acting as independent labs that do testing for a fee. These labs can offer capabilities and expertise that their customer doesn\u2019t have, either because the equipment is specialized and not frequently needed or because the customer is looking for a second opinion on an analysis.\nRegardless of the type of service lab, they all have one thing in common: the way they function. For a moment let\u2019s forget about science and think about something else. Take for example a company that does graphics printing as a service to graphic designers and marketing groups. The company could offer a variety of printing services:\n\n business cards\n stationery (e.g., envelopes, letterhead, etc.)\n postcards\n brochures\n signs\n graphics for trade shows (including mounting of an image on backing, lightboxes, etc.)\n postal marketing services\nIn this scenario, customers can come into the shop and drop off work to be done or place orders online (the company website provides a good description of their services and products). One of their biggest concerns is workflow management: what work is coming in, what is in progress, what is in quality control, and what is ready for delivery. Many activities may be associated with this workflow.\n\n Order acceptance: Log the job into a log book, the go-to reference for work orders. Add the corresponding work order to a binder of work orders; work orders can be removed from the binders as needed (for example, when people are working on the job) and returned when completed. Comments are made on the work order and referenced to an individual\u2019s notebook for details. Work orders shouldn\u2019t be duplicated since people may not be aware of the duplicates and information may be lost. This does add some inefficiency to the process if a work order contains multiple components (e.g., brochures and trade show graphics); if someone needs to work on a task and someone else has the work order, they have to find it. Work orders contain the names of graphics files and their location. Then a check is made to ensure all the needed information is there, notifying people if something is missing. This includes checking to see if the graphics files are available, in the correct format, etc. The priority of the work is determined with respect to other work. Then the customer is notified of the work order status and the expected completion date.\n Scheduling: The work order is assigned to one or more individuals for completion.\n Performing the work: This is where the actual work is performed, including task coordination if an order has multiple components.\n Customer service: This includes answering customer questions about the work order and addressing inquiries on completion date.\n Draft review: This involves obtaining customer sign-off on a prototype stage if required, making adjustments if needed, and then proceeding to completion.\n Quality control: This is where projects are reviewed and approved for completion.\n Delivery: This involves shipping the material back to the customer or notifying the customer the order is ready for pick-up.\n Billing: After satisfaction with the completed work is acknowledged, the work order is billed to the customer.\nWhen the shop has a large number of projects going on, such a manual, paper-based workflow is difficult and time-consuming to manage. Projects have to be scheduled so that they get done and don\u2019t interfere with other projects that might be on a tight deadline. And then there is inventory management, making sure you have the materials you need on hand when you need them. There is also the occasional rescheduling that occurs if equipment breaks down or someone is out sick. A simplified workflow based on the above is shown in Figure 2.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Simplified print shop workflow, with some details omitted for clarity\n\n\n\nLet's say our print shop has seven people working there. The owner manages the overall operation, and an administrator logs in work orders, notes the location of files that will be used on those orders, and does the final checkout of work, shipping, and billing. The remaining five people\u2014staff, although everyone is at the same organizational level\u2014take care of the actual production work; everyone is cross-trained, but some are more adept on some tasks than others.\nImagine you worked in this shop; how might your day go if you were one of the staff? The administrator will have prioritized the work depending on urgency and grouping similar work orders (or partial orders if there is request for multiple services) together. This is just a matter of efficiency: if you are using a particular piece of equipment and it has to be set up, calibrated, and cleaned when finished, you may as well make the most of that effort and run as many similar jobs as you can. Tracking down copies of work orders is an issue if someone is already working part of the order as there is only one copy so that notes and comments don\u2019t get lost. Each staff member has a notebook to keep track of work, any settings used on equipment, and comments about how the work progressed. These notebook entries are important and useful in case questions come up about a job, how it was run, and if any issues were encountered. As one set of jobs is completed, you move on to the next set. Inventory has to be checked to make sure that the needed materials are on-hand or ordered; if something is missing work has to be rescheduled. The workflow is a continual, organized mix of tasks, with people scheduling time on equipment as needed.\nYou can begin to appreciate how difficult the manual, paper-based workflow in a shop like that is to manage, particularly when it depends upon people communicating clearly. It is the same workflow as any service-oriented business, from a florist to a repair shop. What differs is the size of the organization, the complexity of the work, and the education needed to perform the required tasks.\nNow let's get back to the service laboratory. The print shop workflow is much like the structural workflow of such a laboratory. In the end, it\u2019s the nature of the tasks; the complexity of equipment, instrumentation, and electronic systems used; and the education needed to carry out the work that sets the service laboratory apart from other service operations. However, there is one other, critical aspect that sets it apart: most service labs have to meet federal or industry regulations (e.g., the ISO 9000 family of standards) for their operations. \nAs noted earlier, there are many different types of service laboratories. The basic workflow is the same (see Figure 3 for one perspective on the commonalities of research and service laboratories), but the nature of the testing separates one from another. A water testing lab uses different test procedures than a toxicology lab does, or a clinical lab. Those working in different types of labs have to learn how to run different tests, and they also have to learn about the materials they work with. After all, people's observations about the material tested will differ depending upon how much experience they have with different kinds of materials. \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. This diagram represents one perspective on the relationship between laboratory types. This is a bit simplified, particularly on the roles of research labs. Large research facilities, or those in which waiting for test results impacts the progress of research work, may incorporate a \u201cservice lab\u201d function within their operations; the same workflow, just a merger of boxes. The downside of doing that is the loss of independent verification of test results, as people sometimes see what they want to see. This can be addressed by having critical and random samples analyzed independently.\n\n\n\nWhile workflows vary between research and service labs, there is one consistent factor that cuts across both: record keeping.\n\nThe laboratory notebook \nThe laboratory notebook has been a fixture in scientific work for centuries. The laboratory notebook is essentially a diary and can contain text, drawings, pasted photos, illustrations, charts, and so on. Historically, at least until the mid-1970s, it was a paper document that has evolved as legal and regulatory considerations have developed. Figure 4 shows part of Dr. Alexander Graham Bell\u2019s notebook.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Pages 40-41 of Alexander Graham Bell Family Papers in the Library of Congress, Manuscript Division, Public Domain\n\n\n\nThe format of today\u2019s paper notebooks has changed somewhat, and the process of using it has become more rigorous. Take for example Scientific Bindery Productions, a modern manufacturer of professional laboratory notebooks. The description for their duplicate lined notebook includes the following elements[1]:\n\n table of contents\n instructions page, for how to use the notebook and address patent protection\n headers and footers, with legally defensible language\n headers that include title, project number, and book number fields, as well as a \"work continued from page\" section\n footers that include signature, date, disclosed to, and understood by fields, as well as a \"work continued to page\" section\nThe details of some of these points are called out in Figure 5, courtesy of Dr. Raquel Cumeras.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. A lab notebook example, courtesey of Dr. Raquel Cumeras, Science 2 Knowledge blog, 2019\n\n\n\nOver the years, several guidelines have been published about the use of laboratory notebooks. Examples include:\n\n Good manufacturing practice (GMP) and good laboratory practice (GLP) recordkeeping, from David West, St. Louis Community College, Center for Plant and Life Sciences[2]\n NIH scientific recordkeeping guidelines, from the National Institutes of Health[3]\n General laboratory notebook guidelines, from Science editor Elisabeth Pain[4]\nA Google search of \"guidelines for maintaining laboratory notebooks\" or something similar will provide more examples, including those developed by leading universities.\nAt this point, you\u2019re probably wondering why we\u2019re spending so much time on this. The point: good record keeping is the foundation for documenting scientific work regardless of the media, be it paper or electronic. Yes, the laboratory notebook has an electronic equivalent: the electronic laboratory notebook (ELN). These ELNs and other laboratory informatics systems have to support everything paper systems do or they will fail in ensuring the integrity of documented work.\n\nUsing laboratory records \nLaboratory records, whether in laboratory notebooks or some other format, can be acted upon in many ways. Laboratory personnel interact with them by:\n\n recording observations, results, instrument data output, photos, and charts\n describing research processes, goals, and results\n ensuring the authenticity of laboratory work\n planning and collaborating on experiments\n extracting information for reporting\n backing up data\n querying data\n sharing data\n publishing data\n archiving and retrieving data\n securing data\nEverything on that list can be done with paper records; however, those activities are easier, faster, and less error prone with electronic systems. Paper records aren\u2019t going away anytime soon, for example when needing to record comments and information that may not have been provided for in electronic systems. This is particularly true as a project team expands from one person to more people. However, the need to have shared access to information becomes a limiting factor in productivity when we rely on paper-based systems. Paper-based systems also depend upon the proximity of people working together, something that became problematic during the COVID-19 pandemic. Social distancing requirements made sharing paper-based notebook pages more challenging, requiring scanning and emailing. This was perhaps feasible for small amounts of physical materials, but less so for large projects with significant paper-based records.\nThat brings up another important point concerning ownership: whose data is it? When people are handed a notebook, they are told \u201cthis is your notebook, a place to write down your work and observations, and you are responsible for it.\u201d Depending upon how employment or consulting contracts are written, the content that goes into the notebook belongs to whoever is paying for the work. When I worked in a lab, the notebook I used as mine was referenced by others as \u201cyour notebook\u201d (it even had my name on it) even though it wasn\u2019t mine but rather the company\u2019s property. Yet when it was filled, they took possession of it and archived it. This concept of ownership has become a stumbling block in some organizations when they decide to install an ELN or laboratory information management system (LIMS), particularly if there are people who have been working there for a long time and have ingrained behaviors. Those people become concerned that someone is going to see their work in an incomplete state before they\u2019ve reviewed and completed it. It\u2019s their work and they don\u2019t want anyone to look at it until it\u2019s done. While the true owners of the work have always had that right, they may not have exercised it, respecting people\u2019s privacy until the work is complete. If you\u2019re considering an informatics system, does it address that concern about ownership?\n\nBringing informatics into the lab \nSo far this guide has hinted at the implications of adding laboratory informatics systems into the laboratory, but now it's time to discuss it further. Deciding how and when to bring such systems into the lab depends on a number of factors.\n1. What is the lab's budget? If an informatics implementation can't be properly funded, don\u2019t start that implementation until it can be. The cost of a laptop computer is trivial compared to the total cost of implementation.\n2. Do we have in-house access to educated and experienced IT support? That staff should understand that laboratory operations are not just another PC- or Microsoft-dominated arena, but rather an environment which has needs for informatics solutions beyond the typical office solutions. For example, laboratory instruments need to be connected to computers, and the resulting data stores should ideally be integrated to make the data more actionable.\n3. Are laboratory staff ready to use the technologies and take responsibility for the things that go with it? Staff must be trained in more than how to operate an instrument. Can they back up data? Do they understand the security and data privacy risks associated with handling the data?\n4. Are organizational policies flexible enough to allow practical use of the technology while still keeping security and data privacy risks in mind? The organization should have some sort of network access both internally and externally. Remote access should be possible, particularly given the circumstances surrounding pandemics and the like. A balanced policy on taking an organizational laptop out of the laboratory should be in place. Policies on cameras should also be reasonable, allowing researchers to capture images of samples and specimens for their notebooks. If organizational policies are too restrictive, the technology's usefulness is largely overshadowed.\n5. What are the lab's space constraints? The size of a lab and the experiments it must conduct can affect the choice of informatics tools.\n6. What is the nature of the lab's operations? Is it a research lab, service lab, or a combination of both? If you are in a service lab situation, bringing in informatics support as early as possible is essential to your workflow and sanity. You want to minimize having to deal with two separate processes and procedures: the old way we did it (paper-based) and the informatics-supported way.\n7. Is your lab\u2019s operation governed by external regulatory requirements? If it\u2019s going to be in the future, you may as well start as though it currently is. Note that everything should be validated, regardless of whether or not the lab is subject regulations. Validation isn't done to satisfy regulators but rather to prove that a process works properly. If you don\u2019t have that proof, what\u2019s the point of using that process? Do you really want to trust what a process produces without proof that it works?\nMost of the points above are easily understood, but let's go into further detail. Let's start by looking at a simple case of you and your project, where you are the sole contributor, without any need for regulatory oversight. Your primary need is to record your planning notes, observations, results, etc. There are tools within the world of computer software to help with this, most notably the word processor. If you have access to one of those, you probably also have access to spreadsheets and other applications that can make your efforts far easier than working with paper. If you search Google for \u201cword processors as lab notebooks\u201d you will find a number of references, including Dr. Martin Engel's guide to using Microsoft OneNote as an ELN[5] and Labforward's 2020 ELN selection guide.[6]\nHowever, simply switching from paper to electronic doesn't mean you're done. There's more to consider, like developing backup policies, addressing witness review, connecting to instruments, and addressing the effects of team expansion, including expanding to more comprehensive purpose-built informatics solutions.\n\nBackup strategy \nYou have the electronic documentation tools and the skills to use them, but what else do you need? A backup strategy is imperative. Imagine a scenario where you are using a desktop computer, laptop, or tablet to do your work and it has one copy of the document you\u2019ve been working on for weeks. You press the power button one morning and nothing happens. However, you are not (completely) worried or panicked but rather largely calm because:\n\n you have a local backup on removable media (e.g., flash drive, disk), several instances, in fact, that were backed up at least daily, with backups containing everything on the system you were using (you may have a separate backup of your project);\n you have a remote backup on your organization's servers (perhaps on a virtual machine);\n you have a cloud-based backup of at least your project files, and as much of the system files that the cloud storage permits (depending on bandwidth and cost), all secured with two-factor authentication; and\n depending on the operating system you are using, you may have built-in backup-and-recover abilities, e.g. as with Mac OS X's \"Time Machine\" functionality.\nYou've done these things because you've asked yourself \"how much is my work worth to me and my organization?\"\n\nWitness review or sign-off \nThe need for a witness review or sign-off can occur for several reasons, including a potential patent filing or proof that the work was done and data recorded properly, on a certain date, in case it is challenged. One of the ramifications is that you have to identify a second person to be that witness (though this would also be the case if you were using a paper notebook).\nA second issue is that you would have to format the pages of your word processor document (using templates) so as to emulate a signature block and page numbering structure that meets the requirements noted earlier for paper notebooks. You also have to provide a means for either physical (printouts) or electronic signatures (e.g., Adobe Acrobat and other applications provide a useful template that could be used as a lab notebook in this case, doing a cut-and-paste from a Word file). You would also have to ensure that once dated and signed, no edits could be made to that material. If you choose the printed route, then you\u2019re back to paper management. One possibility for dealing with that is to scan the signed pages and upload them to a secure server using the file server's date and time stamp system, or to a document management system to demonstrate that documents haven\u2019t been changed.\nThere is another possibility for time-stamping printed material that is scanned with a high-quality scanner. The concept of machine identification code or \"tracking dots\" allows a set of barely perceptible, forensically traceable colored dots to be printed onto pages, with the arrangement of the dots providing information that identifies the printer (by serial number), as well as the date and time a page was printed. Recent research has demonstrated ways to decode these dots for analysis, particularly as part of printer forensics.[7][8]\n\nInstrument-computer integration \nUsing a computer to host a laboratory notebook raises another concern: how can lab instruments or separate instrument data systems connect to automatically transmit data and information to the computer? Controlled data collection will require software beyond a simple word processor, though many of today's ELN vendors provide integration components with their solutions. This connection, possibly using a spreadsheet import, will greatly improve laboratory productivity and the reliability of data transfers.\n\nExpanding the research team \nIncreasing the size of your project by adding more people will have a significant impact on the complexity of using a more electronic laboratory workflow. However, while the basic issues of sharing, collaboration, etc. are not any different than they would be if paper-based notebooks were in use, the electronic solutions to handling those issues are much more useful. At the core of an expanded laboratory operation is getting personnel to agree on how they are going to communicate data and information, how they are going to collaborate using that data and information, and how they will make that agreement sustainable. There is going to be some small allowance for individual approaches to laboratory activities, but critical matters such as data organization and management need to be strictly adhered to. As such, there are several issues to be mindful of.\n1. How will data and information be organized and compiled? If multiple people are contributing to a project, the results of their work need to be organized in one place so that the status of the work can be ascertained, missing data can be identified, and material is easier to work with. As a project's direction evolves, the formatting and content may change, potentially requiring a complete overhaul of the data structure. That is just a consequence of the way research programs progress.\n2. How will data and information be updated, versioned, and retained? If your lab works with paper files, this isn't so difficult. The lab may have one printed file detailing experimental methods, and when that method file gets updated, the old printed document is removed and the new one added. In labs where each person has their own individual method file, the process would be repeated for each person. As such, there's no confusion as to the current version; methods would have revision histories so that you would know why they were changed. In cases where those methods are kept in electronic files, more attention has to be paid to ensuring that old method files are archived, and that everyone is working with the same version. This means clear communications procedures are essential. Additionally, the name of the file should have the current revision information; don\u2019t rely on the computer's creation or modification dates for files.\n3. How is access to common files controlled? Having people edit or add to common files without careful lockout controls is dangerous. If two people open the same document at the same time and make changes, the one who saves the file last can overwrite any changes the other individual has made. You need a document management system that prevents this; if a file is checked out by one person, others may read it but not write to it until the first person is done. Old versions of files should be archived but not deleted so that a detailed revision history is maintained.\nIf an organization grows too large for a consensus-based cooperation of people using office suite products, it will be time to transition to a more purpose-built solution like a multiuser ELN that is capable of allowing multiple people to contribute to a project simultaneously, providing a common organization for data and information, and allowing users to either import data from an instrument or have a direct connection between the ELN and the instrument data system (or through the use of scientific data management system [SDMS]). But how large is too large? It may be the point when personnel become frustrated with waiting for access to documents, or when processes just don\u2019t seem to move as smoothly as they used to. While there is a significant cost factor in implementing an ELN, it should be done sooner rather than later so your lab can benefit from the organizational structure that an ELN provides and reduce the amount of effort it will take to import data and information into the structure. Ideally you are better off if you can start your work with an ELN once the initial project startup work is taken care of. \nELNs afford a number of additional benefits, including but not limited to access to:\n\n vendor databases for ordering supplies and managing inventory,\n chemical structure databases,\n reaction databases, and\n external libraries.\nELNs are appropriate for both basic research and applied research, including work in testing labs where method development and special project work is being done. The prior bulleted list might give you the impression that ELN are oriented toward chemistry; however, that is a reflection of my experience not the industry. The ELN is used in many other industries. Several examples of ELNs past and present, covering many industries, are listed below.\n\n LabTech Notebook was released in 1986 and discontinued in 2004, and it was designed to provide communications between computers and laboratory instruments that used RS-232 serial communications. This ELN was applicable to a variety of industries and disciplines.\n SmartLab from Velquest was released in the early 2000s and was the first commercial product to carry the \"electronic laboratory notebook\" identifier. It was designed as a platform to encode and guide people conducting lab procedures in GLP\/GMP\/GALP environments. Now owned by Dassault Syst\u00e8mes and rebranded as a laboratory execution system (LES) as part of the BIOVIA product line, the solution's same conceptual functionality has since been incorporated into LIMS and ELNs that fit the more current expectation for an ELN.\n Wolfram Research has as series of notebook products geared toward mathematics.\nThere is a growing list of ELN systems in a variety of disciplines. Given that the scope of activities an ELN can cover is fairly broad, you have to be careful to define your needs before uttering the words \u201cwe\u2019re going to implement an ELN.\u201d We\u2019ll address that point later.\nThe organization of an ELN application is flexible. The layout is user-defined and can contain queryable text, figures, graphics, and fields for instrument-generated data. Because the ELN is inherently flexible and there usually isn\u2019t any quick-start structure, you have to know what you are looking for in a product and how you want to use it. This requires quality due-dilligence research. \u201cIf only I had known\u201d are among the saddest words after product selection.\nSome organizations have chosen to develop their own ELNs. That is something that should be undertaken with fear and trepidation. Part of the original justification for this route is typically based on the belief that you have special needs that can't be met otherwise. Another concern may be that you don\u2019t want to tie yourself to a vendor that may go out of business. Those points are more a matter of product selection criteria than a justification for a software development project. If you do choose to go with an internal or even a contracted development route, you will potentially have to live with a product that has just one customer (unless the contractor decides to productize it, which is an entirely different discussion). You will also be saddled with the support of that product for the rest of its functional life. And that doesn\u2019t even get into the \"who\" of software product management and development, the \u201cwhen will it be done,\u201d and the inevitable scope creep (i.e., the change and expansions of development requirements). In the initial stages of research projects, the organization of data is subject to change as needs change. This is often at the behest of \u201cunstructured\u201d data and the need to manage it. (Note that it isn\u2019t that the data is truly unstructured, it\u2019s just in a variable structure until the project finds its direction.) This can lead to frustration in setting up a commercial ELN system, let alone designing one from scratch. The next section will look at an organization that is more highly structured.\n\nMeeting the needs of the testing laboratory \nThe earlier print shop description will give you an idea of the workflow of a testing or service lab, for example a quality control (QC) lab. In a manually managed QC or research laboratory, operations can become overwhelming quickly. Imagine a study of 40 samples[b] with multiple tests per sample, each of which has to be individually cataloged. Actually I don\u2019t have to imagine it; it happened frequently in our lab, which supported several research groups. When one of these large sample sets shows up\u2014and sometimes more than one\u2014you don't talk to the person doing the sample logging for fear of disrupting their required concentration. You can get them coffee, but no talking.\nWith a LIMS, this isn't so much an issue. You can log in one sample or a hundred, simply by telling it the starting sample ID, how many samples, and what tests should be scheduled. The LIMS then organizes everything and prints the labels for each sample. With some systems, the requestor of a test can even log them in from a web portal, and the central LIMS automatically updates when the samples actually arrive in the lab. \nA LIMS makes life easier for laboratories in a number of other ways as well. Want to find a list of samples that are pending a particular test? A quality LIMS can readily display that information, including the sample numbers, priorities, and current locations, with no need to manually check work request sheets. Does a third party want to find out the status of one or more of their in-process samples? Role-based access management means a third party can receive limited access to view that status, without seeing anyone else's sensitive data. What about verifying and approving results? The LIMS can provide some level of results checking, with final verification and approval by lab management. When approved, the reports for each set of requests can be printed, emailed, or stored for portal access. And what about integrating data and systems? The LIMS can be connected to an instrument data system (IDS). Depending on the sophistication of that system, the LIMS can generate a worklist of samples that needs to be processed by that device, with the list downloaded to the IDS. When the work is completed, the results can be uploaded directly to the LIMS. This type of system interaction is one of the places where significant productivity gains can be had.\nThe key attributes of a LIMS are shown in Figure 6.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. Slide detailing the core components of a LIMS, from A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work, Part 1: Laboratory Informatics Technologies, a webinar series by Joe Liscouski\n\n\n\nThese are just the highlights of what a LIMS can do. Why is a LIMS more effective out-of-the-box than an ELN? The operational behavior of testing labs is essentially the same across industries and disciplines, and as a result vendors have a more predictable and stable base of customers and user requirements to work against than an ELN. It\u2019s a bit like comparing home building contractors. Some have a basic structural architecture that they can modify to meet the needs of a broad market. Others do completely custom homes from the basement up, each of which is unique. The economies of scale and a broader, more predictable customer base show up in products that are easier to work with and adapt both for the primary vendor and those providing add-ons. Those LIMS add-ons include specialized facilities for different industries, including enology and viticulture analysis, water treatment, mineral analysis, cannabis testing, and clinical diagnostics (though the system used in the clinical setting is typically referred to as a laboratory information system or LIS). Regardless, and as is the case with ELNs, you want to install a LIMS as soon as you are able to avoid issues surrounding differing workflows based on pre- and post-implementation.\n\nOther laboratory informatics systems \nWe've mentioned a few other systems in passing, but here we'll provide a brief overview of a few of them.\n\nScientific data management system (SDMS) \nThe SDMS helps laboratories better solve the problem of dealing with a large number of data files that are being generated, basically by acting as a giant file cabinet that LIMS, ELN, and IDS can connect to. For example, you may not want to put large datasets, spectra, etc. in a LIMS or ELN, you but still have to reference those large files within other internal LIMS and ELN files. This is where the SDMS steps in, storing those large files in the \"file cabinet\" while maintaining references and metadata that are usable by other informatics systems.\n\nLaboratory execution system (LES) \nYou may not know about them, but there are lesser-known systems called LES that are designed to ensure testing protocols are executed properly and that all the data that is necessary is properly recorded. The initial incarnation arose from the previously discussed SmartLab[c], a stand-alone product that would guide an analyst through the steps of an analysis, recording each detail of the work in a file that regulatory agencies could inspect to ensure that work was done properly. It found a ready market in any lab that needed to meet GLP\/GMP requirements. The functionality needed to create the same capability can be found in some LIMS and ELNs, but programming is required.\n\nInstrument data system (IDS) \nAny laboratory instrument popular in the marketplace has had a computer either attached to it or built into it as a package. That\u2019s what an IDS is. It provides automated control over an instrument, collecting and analyzing the data produced by the measuring components. Depending on the sophistication of the vendor, and the demands of the marketplace, the connections between the IDS and another laboratory informatics solution may range from user-programmable interfaces (via a network, USB, serial, or digital I\/O connection) to built-in communications systems that are almost plug-and-play. The latter are most commonly found in the clinical chemistry market, where a great deal of attention has been paid to integration and systems communication via Health Level 7 (HL7) and related protocols. (The details, however, are beyond the scope of this document.)\n\nPlanning for laboratory informatics \nThere are two key requirements to the successful implementation of informatics products: education and planning. \n\nEducation \nAs far as education goes, the webinar series noted in Figure 6 is a good tool, as are documents provided by technical standards body ASTM International. ASTM documents that may be of value to you include:\n\n ASTM E1578-18 Standard Guide for Laboratory Informatics[9]\n ASTM E1578-13 Standard Guide for Laboratory Informatics[10] (contains some information not found in 2018 version)\n ASTM E1947-98(2014) Standard Specification for Analytical Data Interchange Protocol for Chromatographic Data[11]\n ASTM D8244-21 Standard Guide for Analytical Laboratory Operations Supporting the Cannabis\/Hemp Industry[12]\n If you search the ASTM website for \"LIMS\" or \"informatics,\" you'll be surprised by the amount of other material that shows up.\nOther sources of educational material include:\n\n The Tutorial section of LIMSforum.com[13]\n Laboratory Information Systems Project Management: A Guidebook for International Implementations, by the APHL\n Laboratory Informatics Guide 2021, by Scientific Computing World[14]\n Lab Manager magazine, published by LabX Media Group[15]\n Computerized Systems in the Modern Laboratory: A Practical Guide, by Joe Liscouski[16]\n Any vendor with an informatics product will send you a nearly endless stream of material.\nUltimately, however, the responsibility for informatics implementation projects doesn't exist solely on the shoulders of laboratory personnel. Everyone connected with a laboratory implementing informatics solutions should have some level of awareness regarding laboratory informatics, including upper management. However, the level of knowledge required may vary slightly depending on the role. For example, laboratory personnel should be fully educated on common laboratory technologies at a minimum. They should also understand why an informatics project is being considered, what the scope of the implementation will be, and what their role will be in the implementation. Upper management should remember to ask laboratory personnel for input on the project, including the topic of product requirements, in order for personnel to feel like they are part of the process, not simply an observer or \"something that's happening to them.\" Finally, laboratory personnel should understand how their jobs are going to change once the implementation is complete. This needs to be addressed very early in project discussions; it is a matter of change, and change scares people, particularly if it affects their job and income. This is not a \u201cwe\u2019ll deal with that later\u201d point. Don\u2019t start the discussion until you figure this out. Things may change, but people want security.\nOf course, any information technology (IT) personnel will also be involved, requiring significant knowledge about not only networking and software installations but also systems integrations. IT personnel need to understand their role in the implementation project, which can include support, project management, evaluating the vendor support capabilities, and more. They should also be fully aware of and understand the technologies the organization is considering for implementation, and they will be a vital part of the project planning and vendor selection process. IT personnel also will be interested in questions about any enterprise resource planning (ERP) aspects, which may raise issues of \"build or buy.\" The organization needs to be prepared to both address these concerns and gain IT personnel as strong supporters of the project.\nFinally, upper management\u2014those who are going to approve the project and provide funding\u2014need to be educated enough to understand the benefits and risks of the proposed implementation, including the likely time scale for the work. Upper management will need to be active in the project in at least two critical junctures, plus at specific milestones as needed. The first time upper management will need informed participation will be during initial project planning. They will help the organization lay out the issues that need to be addressed, the scope of options that will be investigated, and how the organization is going to proceed. They may pose questions such as \u201ccan we use existing system to solve the problem,\u201d particularly if there already has been an investment in an ERP solution such as SAP. Such technology questions will also be of interest to IT personnel since they have an investment in those systems. The second time upper management needs to undoubtedly be involved is when the actual project proposal is finished and is ready to be pitched. They will ask need to ask questions about the reasoning behind the choices made, why current systems are insufficient, what kind of investment the project will require, how the implementation will be scheduled, and how the roll-out would be planned. Understanding the answers to these and other questions will be difficult if upper management doesn\u2019t understand the technology, the issues, the options, and the benefits of the proposed laboratory informatics project.\nIf the world of informatics is new to any or all these stakeholders, the organization must consider getting outside support (not from a vendor) to guide the organization through the process of evaluating the lab's operations, scoping out an overall implementation program, dividing it into milestones, setting priorities, and developing user requirements.\n\nPlanning \nThe whole point of project planning is to get your organization from the starting point to the end goal, preferably using the most effective path. So, where is the organization going, and why? Everything pretty much boils down to those two questions. Once those questions are answered, more will arise concerning how the organization is actually going to get to the project's end goal.\nInitial planning will look at the lab from a standpoint of the minimum number of computers required to do the work. In some cases, perhaps just those computational systems built into instruments will be sufficient. Whatever the decision, that\u2019s the planning baseline. Then consider what kind of lab it is: a research lab, a service lab, or a blend? That helps direct concentration on potential solutions; however, be sure not to completely eliminate other options just yet. For example, if your lab is a QC lab, it's probably a service lab in need of a LIMS, but even then there are still options to evaluate.\nFrom there, the organization must also think in terms of where that baseline lab is going to be in five years; it\u2019s an arbitrary number but a starting point. Why that far out? It will take a year or two to plan, implement, validate, and educate people to work with the new systems and for the lab to change its behavior and settle down to a new mode of working. The organization should also consider what other changes are likely to take place. What new equipment, procedures, and management changes can be anticipated? Ideally any implemented informatics system will be in place and stable for a few years before people start asking for significant changes; minor ones will likely happen early on. Any hint of \u201cwe didn\u2019t plan for that\u201d will be viewed as poor leadership. Figure 7 shows some of the key points you need to look at during planning.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 7. Slide detailing considerations of laboratory informatics implementation in the lab, from A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work, Part 4: LIMS\/LIS, ELN, SDMS, IT & Education, a webinar series by Joe Liscouski\n\n\n\nAnother factor that needs to be considered is that the considerations shown in Figure 7 can be repeated for each lab in the organization. The project planning team also needs to consider how current laboratory workflow impacts other labs. Are there synergistic effects that can be gained by broadening the scope of what the lab is doing?\n\nWhy projects fail \nWe shouldn't finish the planning section of this guide without discussing why laboratory informatics implementation projects can and do fail. These types of projects are large software projects, and delays, cost over-runs, and failures are common (but hopefully not for your organization's project). Just Google \"why IT projects fail\" and read through some anecdotes. The following are some common reasons informatics implementation projects fail.\n\n Insufficient budgeting: Projects can run short of funding, requiring an awkward meeting with management asking for additional funding (without a project change to account for it), inevitably showing a lack of foresight and planning. Build in a large contingency fund because the unexpected happens. If you\u2019d like some education on the topic, watch a few episodes of any home upgrade project on HGTV, in particular Love It or List It.[d]\n Insufficient management support: If sufficient communication isn\u2019t made with management, problems may arise. For example, project delays are a fact of life. Keep clear communications with upper management so that they, and everyone else on the project, know what is going on. Miscommunication or lack of communication of other aspects of the project may inevitably doom the project.\n Poor understanding of the scope and nature of the project: This is an educational issue, and a lack of education for all involved parties is almost a guarantee of failure. If you need help, bring in an independent consultant who can lend confidence to the project and its management.\n Lack of quality or attention to detail: \u201cThere is always time to do it over, but not enough to do it right\u201d is a common complaint in engineering projects. If you hear it on your organization's project, you are in trouble. Basically the complaint is that project members are cutting corners and not doing things properly, and in a way that is not supportable. This never ends well; sometimes not quickly, but in the long run it leads to problems. \n Poor or unrealistic timelines: You may as well face reality from the start: an aggressive timeline just leads to problems (see bullet above). Timelines expand, but they almost never get shorter. If the project team is always rushing, something will most certainly get missed, causing problems later down the road.\n Poor project management: Well-managed projects are obvious, and so are poorly managed ones. Just watch the people working, their demeanor, and their attitude about the work; it will tell you all you need to know. Well-managed projects may not always run smoothly, but they make consistent progress. Poorly managed projects cause you to make excuses.\nClosing \nThis guide is like the directory signs in shopping malls: they tell you where you are and what shops and restaurants in the facility to consider. Once you figure out what you are looking for, you can find your way there. Hopefully in reading this you\u2019ve formed an idea of what you want to look at and what your path to finding it is.\n\nFootnotes \n\n\n\u2191 According to Greek mythology (from the E2BN Myths page): \"Long long ago, when Queen Athena (Zeus's daughter) was born, Zeus blessed her with two boons for when she came of age. After almost 15 years, Athena was told to think up two things to ask for ... 1) To have a city in Greece named after her (Athens) [and] 2) To have all the people of the world see her face every day of the year (what you are seeing are only her eyes). Thus, the sky is blue, just like the color of Athena's eyes...\" \n\n\u2191 In some life science drug screening studies, the number can be far higher, which is where robotics and automation becomes important. \n\n\u2191 A name that has been used by conference organizers after the product was sold and renamed. If you do a Google search on \u201cSmartLab,\u201d you may be surprised at what turns up. \n\n\u2191 Some season highlights can be found on the HGTV website. \n\n\nAbout the author \nInitially educated as a chemist, author Joe Liscouski (joe dot liscouski at gmail dot com) is an experienced laboratory automation\/computing professional with over forty years of experience in the field, including the design and development of automation systems (both custom and commercial systems), LIMS, robotics and data interchange standards. He also consults on the use of computing in laboratory work. He has held symposia on validation and presented technical material and short courses on laboratory automation and computing in the U.S., Europe, and Japan. He has worked\/consulted in pharmaceutical, biotech, polymer, medical, and government laboratories. His current work centers on working with companies to establish planning programs for lab systems, developing effective support groups, and helping people with the application of automation and information technologies in research and quality control environments.\n\nReferences \n\n\n\u2191 \"Duplicate Lined Notebook\". Scientific Bindery Productions, Inc. https:\/\/scientificbindery.com\/products\/duplicate-lined-notebook\/ . Retrieved 12 May 2021 .   \n\n\u2191 West, D. (n.d.). \"GMP\/GLP Recordkeeping\". St. Louis Community College, Center for Plant and Life Sciences. https:\/\/users.stlcc.edu\/departments\/fvbio\/Lab_Practices_GLP_STLCC.htm . Retrieved 12 May 2021 .   \n\n\u2191 National Institutes of Health (December 2008). \"Guidelines for Scientific Record Keeping in the Intramural Research Program at the NIH\" (PDF). Office of the Director. https:\/\/oir.nih.gov\/sites\/default\/files\/uploads\/sourcebook\/documents\/ethical_conduct\/guidelines-scientific_recordkeeping.pdf . Retrieved 12 May 2021 .   \n\n\u2191 Pain, E. (03 September 2019). \"How to keep a lab notebook\". Science. doi:10.1126\/science.caredit.aaz3678. https:\/\/www.sciencemag.org\/careers\/2019\/09\/how-keep-lab-notebook . Retrieved 12 May 2021 .   \n\n\u2191 Engel, M. (December 2015). \"Blog: How to use onenote as your electronic lab book\". MartinEngel.net. http:\/\/martinengel.net\/2015\/12\/how-to-use-onenote-as-your-electronic-notebook\/ . Retrieved 12 May 2021 .   \n\n\u2191 Bungers, S. (2020). \"The Electronic Lab Notebook in 2020: A comprehensive guide\". Labforward GmbH. https:\/\/www.labfolder.com\/electronic-lab-notebook-eln-research-guide . Retrieved 12 May 2021 .   \n\n\u2191 Richter, T.; Escher, S.; Sch\u00f6nfeld, D. et al. (2018). \"Forensic Analysis and Anonymisation of Printed Documents\". Proceedings of the 6th ACM Workshop on Information Hiding and Multimedia Security: 127\u201338. doi:10.1145\/3206004.3206019.   \n\n\u2191 Baraniuk, C. (07 June 2017). \"Why printers add secret tracking dots\". BBC Future. Archived from the original on 02 November 2019. https:\/\/web.archive.org\/web\/20191102031255\/https:\/\/www.bbc.com\/future\/article\/20170607-why-printers-add-secret-tracking-dots . Retrieved 13 May 2021 .   \n\n\u2191 \"ASTM E1578-18 Standard Guide for Laboratory Informatics\". ASTM International. 2018. doi:10.1520\/E1578-18. https:\/\/www.astm.org\/Standards\/E1578.htm . Retrieved 13 May 2021 .   \n\n\u2191 \"ASTM E1578-13 Standard Guide for Laboratory Informatics\". ASTM International. 2013. doi:10.1520\/E1578-13. https:\/\/www.astm.org\/DATABASE.CART\/HISTORICAL\/E1578-13.htm . Retrieved 13 May 2021 .   \n\n\u2191 \"ASTM E1947-98(2014) Standard Specification for Analytical Data Interchange Protocol for Chromatographic Data\". ASTM International. 2014. doi:10.1520\/E1947-98R14. https:\/\/www.astm.org\/Standards\/E1947.htm . Retrieved 13 May 2021 .   \n\n\u2191 \"ASTM D8244-21 Standard Guide for Analytical Laboratory Operations Supporting the Cannabis\/Hemp Industry\". ASTM International. 2021. doi:10.1520\/D8244-21. https:\/\/www.astm.org\/Standards\/D8244.htm . Retrieved 13 May 2021 .   \n\n\u2191 \"Tutorials\". LIMSforum. LabLynx, Inc. https:\/\/www.limsforum.com\/category\/education\/tutorials-education\/ . Retrieved 13 May 2021 .   \n\n\u2191 \"Laboratory Informatics Guide 2021\". Scientific Computing World. 2021. https:\/\/www.scientific-computing.com\/issue\/laboratory-informatics-guide-2021 . Retrieved 13 May 2021 .   \n\n\u2191 \"Lab Manager\". LabX Media Group. 2021. https:\/\/www.labmanager.com\/magazine . Retrieved 13 May 2021 .   \n\n\u2191 Liscouski, J.G. (2015). Computerized Systems in the Modern Laboratory: A Practical Guide. DHI Publishing. pp. 432. ISBN 9781933722863. https:\/\/www.dhibooks.com\/computerized-systems-in-the-modern-laboratory-a-practical-guide .   \n\n\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\">https:\/\/www.limswiki.org\/index.php\/LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies<\/a>\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tLII\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 14 May 2021, at 17:31.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 436 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","d8b467af534a70312a21f63b61be26cd_html":"<body class=\"mediawiki ltr sitedir-ltr ns-202 ns-subject page-LII_The_Application_of_Informatics_to_Scientific_Work_Laboratory_Informatics_for_Newbies skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">LII:The Application of Informatics to Scientific Work: Laboratory Informatics for Newbies<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\"><p><b>Title<\/b>: <i>The Application of Informatics to Scientific Work: Laboratory Informatics for Newbies<\/i>\n<\/p><p><b>Author for citation<\/b>: Joe Liscouski, with editorial modifications by Shawn Douglas\n<\/p><p><b>License for content<\/b>: <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/creativecommons.org\/licenses\/by-sa\/4.0\/\" target=\"_blank\">Creative Commons Attribution-ShareAlike 4.0 International<\/a>\n<\/p><p><b>Publication date<\/b>: April 2021\n<\/p>\n\n\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>The purpose of this piece is to introduce people who are not intimately familiar with <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> work to the basics of laboratory operations and the role that <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics_(academic_field)\" title=\"Informatics (academic field)\" class=\"wiki-link\" data-key=\"0391318826a5d9f9a1a1bcc88394739f\">informatics<\/a> can play in assisting scientists, engineers, and technicians in their efforts. The concepts are important because they provide a functional foundation for understanding lab work and how that work is done in the early part of the twenty-first century (things will change, just wait for it).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Intended_audience\">Intended audience<\/span><\/h3>\n<p>This material is intended for anyone who is interested in seeing how modern informatics tools can help those doing scientific work. It will provide an orientation to scientific and laboratory work, as well as the systems that have been developed to make that work more productive. It\u2019s for people coming out of school who have carried out lab experiments but not corporate research projects, for those who need to understand how testing labs work, and for IT professionals who may be faced with supporting computing systems in lab environments. It\u2019s also for those who may be tasked with managing projects to choose, install, and make informatics tools useful.\n<\/p><p>Figure 1 shows the elements we\u2019ll be discussing in this piece. The treatment of the technical material will be on the lighter side, leaving in-depth subject matter to other works. Instrument data systems will be covered lightly, as any serious discussion becomes lengthy and discipline-specific very quickly; additionally, that material has been covered <a href=\"https:\/\/www.limswiki.org\/index.php\/LII:Notes_on_Instrument_Data_Systems\" title=\"LII:Notes on Instrument Data Systems\" class=\"wiki-link\" data-key=\"1b7330228fd59158aab6fab82ad0e7cc\">in other works<\/a>.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Liscouski_AppInfoSciWork21.png\" class=\"image wiki-link\" data-key=\"fd326d7208c88d3103f5af9e4a306269\"><img alt=\"Fig1 Liscouski AppInfoSciWork21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/5c\/Fig1_Liscouski_AppInfoSciWork21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Elements we\u2019ll be covering<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Types_of_scientific_and_laboratory_work\">Types of scientific and laboratory work<\/span><\/h2>\n<p>Science is about seeking truthful answers to questions. Sometimes those questions are open-ended without any idea where they will lead you in answering them (e.g. \u201cWhy does water ice float?\u201d). Others are very specific, concerning material composition or properties (e.g., \u201cHow much lead is in this drinking water?\u201d, \u201cHow much does a butterfly weigh?\u201d). Still others may take some effort before you determine the best approach to working on them. The approach someone uses to address these questions depends on the nature of the question; some are destined for research, while others are addressed using specific test methods.\n<\/p><p>There are two types of research: basic and applied. Both can include field work, observations, experiments, models (mathematical, computer, and simulation), etc. Applied research is also done in testing or service laboratories, as with, for example, the development of new methods of analysis.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Basic_and_applied_research\">Basic and applied research<\/span><\/h3>\n<p>Basic research is open-ended, as you are looking into something without any idea of where the work will lead. It is often funded by grants through universities or government institutions; continued support depends on the perceived value of the research. Projects can range in size from the work of a single individual to a small team to large-scale groups studying astronomy, high-energy physics, engineering, the life sciences, or a number of fields.\n<\/p><p>Applied research, on the other hand, is directed toward a goal. That goal could be a cure for a disease, the development of a <a href=\"https:\/\/www.limswiki.org\/index.php\/COVID-19\" title=\"COVID-19\" class=\"mw-redirect wiki-link\" data-key=\"da9bd20c492b2a17074ad66c2fe25652\">COVID-19<\/a> vaccine, or work towards <a href=\"https:\/\/www.limswiki.org\/index.php\/Artificial_intelligence\" title=\"Artificial intelligence\" class=\"wiki-link\" data-key=\"0c45a597361ca47e1cd8112af676276e\">artificial intelligence<\/a> (AI). As with basic research, until some early goals have been reached the work may begin with a single individual or a small team, and then the project scales up. The effort may be broken down into a set of more narrowly focused efforts, whose results will be combined as the development proceeds. Since applied research is goal-directed, funding will depend upon who benefits from those goals being met. Projects of national interest, including security, may be wholly or partially funded by the government. Projects with a commercial interest tend to be funded by corporate interests, including individual companies in their own laboratories or through contract research organizations with expertise useful to the program. Where there is interest from a number of corporate and\/or government groups, consortiums may form to distribute the cost and share in the results.\n<\/p><p>Both basic and applied research can be found in government institutions (including military groups, research and development agencies like the Defense Advanced Research Project Agency [DARPA], and task-specific institutions such as the <a href=\"https:\/\/www.limswiki.org\/index.php\/National_Institutes_of_Health\" title=\"National Institutes of Health\" class=\"wiki-link\" data-key=\"e5c215c48e73ae58b0695dc2af951cd0\">National Institutes of Health<\/a> [NIH]), public and private non-profit groups, corporations, consortia, and contract research organizations.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"The_research_process\">The research process<\/span><\/h4>\n<p>The research process begins with a question. Any question will do, including \u201cwhy is the sky blue?\u201d We\u2019ll bypass Greek mythology<sup id=\"rdp-ebb-cite_ref-1\" class=\"reference\"><a href=\"#cite_note-1\">[a]<\/a><\/sup> by asking more questions and planning how to proceed to answer them. For example, \u201cIs the sky always blue?\u201d, \u201cWhen is\/isn\u2019t it?\u201d, and \u201cWhat other colors can it be?\u201d Once the process begins, it can include a number of steps, the choice and direction depending upon the nature of the research and the mindset of the researcher:\n<\/p>\n<ul><li> Observations: This includes basic note-taking with support material (text, photos, drawings, charts, and scanned material). Research (e.g., as with basic astronomy, field biology) can be as simple as looking something up on Google or as complex as understanding how a virus works. Research is about asking questions and looking for answers, which often leads to more questions. It\u2019s a little like my granddaughter who always asks \u201cwhy?\u201d no matter how well I answer the previous question (or at least how well I think I did).<\/li>\n<li> Enhanced observations: This includes interacting with items under observation, as well as non-directed interactions, preliminary data gathering, and behavioral analysis.<\/li>\n<li> Experiments and information gathering: This includes organized experiments that are planned, directed, and purpose-driven, as well as data and information gathering.<\/li>\n<li> Team building: This includes the creation of teams or networks of people working on the same or similar projects.<\/li>\n<li> Analytics and reporting: This includes data and information analysis, data modeling (e.g., mathematical, computer algorithm, and simulation), information synthesis, and knowledge creation.<\/li>\n<li> Technology acquisition: This includes gaining access to public, commercial, remote, and other types of databases to assist the research.<\/li><\/ul>\n<p>Pinning down a \u201ctypical\u201d approach to research isn\u2019t possible because the routes people follow are as individual as the researchers and their area of work are. However, this is generally not the case with testing labs.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Testing_laboratories\">Testing laboratories<\/span><\/h2>\n<p>In addition to research labs, there are also testing or \"service\" laboratories. Service labs carry out specific test routines on samples and specimens; you may be familiar with them as <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> labs, <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_laboratory\" title=\"Clinical laboratory\" class=\"wiki-link\" data-key=\"307bcdf1bdbcd1bb167cee435b7a5463\">clinical labs<\/a>, toxicology labs, <a href=\"https:\/\/www.limswiki.org\/index.php\/Forensic_laboratory\" title=\"Forensic laboratory\" class=\"mw-redirect wiki-link\" data-key=\"dabef2566b55c1f13b395543b47ae81e\">forensic science labs<\/a>, etc. They are called service labs because they support other organizations, including research organizations, and they have similar modes of operation and work organization, running different tests depending on their area of specialization.\n<\/p><p>Contract testing labs are another flavor of service laboratories, acting as independent labs that do testing for a fee. These labs can offer capabilities and expertise that their customer doesn\u2019t have, either because the equipment is specialized and not frequently needed or because the customer is looking for a second opinion on an analysis.\n<\/p><p>Regardless of the type of service lab, they all have one thing in common: the way they function. For a moment let\u2019s forget about science and think about something else. Take for example a company that does graphics printing as a service to graphic designers and marketing groups. The company could offer a variety of printing services:\n<\/p>\n<ul><li> business cards<\/li>\n<li> stationery (e.g., envelopes, letterhead, etc.)<\/li>\n<li> postcards<\/li>\n<li> brochures<\/li>\n<li> signs<\/li>\n<li> graphics for trade shows (including mounting of an image on backing, lightboxes, etc.)<\/li>\n<li> postal marketing services<\/li><\/ul>\n<p>In this scenario, customers can come into the shop and drop off work to be done or place orders online (the company website provides a good description of their services and products). One of their biggest concerns is <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a> management: what work is coming in, what is in progress, what is in quality control, and what is ready for delivery. Many activities may be associated with this workflow.\n<\/p>\n<ul><li> Order acceptance: Log the job into a log book, the go-to reference for work orders. Add the corresponding work order to a binder of work orders; work orders can be removed from the binders as needed (for example, when people are working on the job) and returned when completed. Comments are made on the work order and referenced to an individual\u2019s notebook for details. Work orders shouldn\u2019t be duplicated since people may not be aware of the duplicates and information may be lost. This does add some inefficiency to the process if a work order contains multiple components (e.g., brochures and trade show graphics); if someone needs to work on a task and someone else has the work order, they have to find it. Work orders contain the names of graphics files and their location. Then a check is made to ensure all the needed information is there, notifying people if something is missing. This includes checking to see if the graphics files are available, in the correct format, etc. The priority of the work is determined with respect to other work. Then the customer is notified of the work order status and the expected completion date.<\/li>\n<li> Scheduling: The work order is assigned to one or more individuals for completion.<\/li>\n<li> Performing the work: This is where the actual work is performed, including task coordination if an order has multiple components.<\/li>\n<li> Customer service: This includes answering customer questions about the work order and addressing inquiries on completion date.<\/li>\n<li> Draft review: This involves obtaining customer sign-off on a prototype stage if required, making adjustments if needed, and then proceeding to completion.<\/li>\n<li> Quality control: This is where projects are reviewed and approved for completion.<\/li>\n<li> Delivery: This involves shipping the material back to the customer or notifying the customer the order is ready for pick-up.<\/li>\n<li> Billing: After satisfaction with the completed work is acknowledged, the work order is billed to the customer.<\/li><\/ul>\n<p>When the shop has a large number of projects going on, such a manual, paper-based workflow is difficult and time-consuming to manage. Projects have to be scheduled so that they get done and don\u2019t interfere with other projects that might be on a tight deadline. And then there is inventory management, making sure you have the materials you need on hand when you need them. There is also the occasional rescheduling that occurs if equipment breaks down or someone is out sick. A simplified workflow based on the above is shown in Figure 2.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Liscouski_AppInfoSciWork21.png\" class=\"image wiki-link\" data-key=\"e6a7aef6488ffa2faf96dfaaa770b627\"><img alt=\"Fig2 Liscouski AppInfoSciWork21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/14\/Fig2_Liscouski_AppInfoSciWork21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Simplified print shop workflow, with some details omitted for clarity<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Let's say our print shop has seven people working there. The owner manages the overall operation, and an administrator logs in work orders, notes the location of files that will be used on those orders, and does the final checkout of work, shipping, and billing. The remaining five people\u2014staff, although everyone is at the same organizational level\u2014take care of the actual production work; everyone is cross-trained, but some are more adept on some tasks than others.\n<\/p><p>Imagine you worked in this shop; how might your day go if you were one of the staff? The administrator will have prioritized the work depending on urgency and grouping similar work orders (or partial orders if there is request for multiple services) together. This is just a matter of efficiency: if you are using a particular piece of equipment and it has to be set up, calibrated, and cleaned when finished, you may as well make the most of that effort and run as many similar jobs as you can. Tracking down copies of work orders is an issue if someone is already working part of the order as there is only one copy so that notes and comments don\u2019t get lost. Each staff member has a notebook to keep track of work, any settings used on equipment, and comments about how the work progressed. These notebook entries are important and useful in case questions come up about a job, how it was run, and if any issues were encountered. As one set of jobs is completed, you move on to the next set. Inventory has to be checked to make sure that the needed materials are on-hand or ordered; if something is missing work has to be rescheduled. The workflow is a continual, organized mix of tasks, with people scheduling time on equipment as needed.\n<\/p><p>You can begin to appreciate how difficult the manual, paper-based workflow in a shop like that is to manage, particularly when it depends upon people communicating clearly. It is the same workflow as any service-oriented business, from a florist to a repair shop. What differs is the size of the organization, the complexity of the work, and the education needed to perform the required tasks.\n<\/p><p>Now let's get back to the service laboratory. The print shop workflow is much like the structural workflow of such a laboratory. In the end, it\u2019s the nature of the tasks; the complexity of equipment, instrumentation, and electronic systems used; and the education needed to carry out the work that sets the service laboratory apart from other service operations. However, there is one other, critical aspect that sets it apart: most service labs have to meet federal or industry regulations (e.g., the <a href=\"https:\/\/www.limswiki.org\/index.php\/ISO_9000\" title=\"ISO 9000\" class=\"wiki-link\" data-key=\"53ace2d12e80a7d890ce881bc6fe244a\">ISO 9000<\/a> family of standards) for their operations. \n<\/p><p>As noted earlier, there are many different types of service laboratories. The basic workflow is the same (see Figure 3 for one perspective on the commonalities of research and service laboratories), but the nature of the testing separates one from another. A water testing lab uses different test procedures than a toxicology lab does, or a clinical lab. Those working in different types of labs have to learn how to run different tests, and they also have to learn about the materials they work with. After all, people's observations about the material tested will differ depending upon how much experience they have with different kinds of materials. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Liscouski_AppInfoSciWork21.png\" class=\"image wiki-link\" data-key=\"ee73859dfa7a4ff773b8022433c7c3d9\"><img alt=\"Fig3 Liscouski AppInfoSciWork21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/62\/Fig3_Liscouski_AppInfoSciWork21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> This diagram represents one perspective on the relationship between laboratory types. This is a bit simplified, particularly on the roles of research labs. Large research facilities, or those in which waiting for test results impacts the progress of research work, may incorporate a \u201cservice lab\u201d function within their operations; the same workflow, just a merger of boxes. The downside of doing that is the loss of independent verification of test results, as people sometimes see what they want to see. This can be addressed by having critical and random samples analyzed independently.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>While workflows vary between research and service labs, there is one consistent factor that cuts across both: record keeping.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_laboratory_notebook\">The laboratory notebook<\/span><\/h2>\n<p>The <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_notebook\" title=\"Laboratory notebook\" class=\"wiki-link\" data-key=\"be60c7be96aba8e9a84537fd8835fa54\">laboratory notebook<\/a> has been a fixture in scientific work for centuries. The laboratory notebook is essentially a diary and can contain text, drawings, pasted photos, illustrations, charts, and so on. Historically, at least until the mid-1970s, it was a paper document that has evolved as legal and regulatory considerations have developed. Figure 4 shows part of Dr. Alexander Graham Bell\u2019s notebook.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:AGBell_Notebook.jpg\" class=\"image wiki-link\" data-key=\"ba0c4bed9e7d80a0155a84a8d79159bf\"><img alt=\"AGBell Notebook.jpg\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/0\/0c\/AGBell_Notebook.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Pages 40-41 of Alexander Graham Bell Family Papers in the Library of Congress, Manuscript Division, Public Domain<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The format of today\u2019s paper notebooks has changed somewhat, and the process of using it has become more rigorous. Take for example Scientific Bindery Productions, a modern manufacturer of professional laboratory notebooks. The description for their duplicate lined notebook includes the following elements<sup id=\"rdp-ebb-cite_ref-SBPDuplicate_2-0\" class=\"reference\"><a href=\"#cite_note-SBPDuplicate-2\">[1]<\/a><\/sup>:\n<\/p>\n<ul><li> table of contents<\/li>\n<li> instructions page, for how to use the notebook and address patent protection<\/li>\n<li> headers and footers, with legally defensible language<\/li>\n<li> headers that include title, project number, and book number fields, as well as a \"work continued from page\" section<\/li>\n<li> footers that include signature, date, disclosed to, and understood by fields, as well as a \"work continued to page\" section<\/li><\/ul>\n<p>The details of some of these points are called out in Figure 5, courtesy of Dr. Raquel Cumeras.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Liscouski_AppInfoSciWork21.png\" class=\"image wiki-link\" data-key=\"2cb81832fb7493dc7d899dce7df6a005\"><img alt=\"Fig5 Liscouski AppInfoSciWork21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/65\/Fig5_Liscouski_AppInfoSciWork21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/science2knowledge.wordpress.com\/writing-the-laboratory-notebook\/\" target=\"_blank\">A lab notebook example<\/a>, courtesey of Dr. Raquel Cumeras, <i>Science 2 Knowledge<\/i> blog, 2019<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Over the years, several guidelines have been published about the use of laboratory notebooks. Examples include:\n<\/p>\n<ul><li> Good manufacturing practice (GMP) and good laboratory practice (GLP) recordkeeping, from David West, St. Louis Community College, Center for Plant and Life Sciences<sup id=\"rdp-ebb-cite_ref-WestGMP_3-0\" class=\"reference\"><a href=\"#cite_note-WestGMP-3\">[2]<\/a><\/sup><\/li>\n<li> NIH scientific recordkeeping guidelines, from the National Institutes of Health<sup id=\"rdp-ebb-cite_ref-NIHGuidelines08_4-0\" class=\"reference\"><a href=\"#cite_note-NIHGuidelines08-4\">[3]<\/a><\/sup><\/li>\n<li> General laboratory notebook guidelines, from <i>Science<\/i> editor Elisabeth Pain<sup id=\"rdp-ebb-cite_ref-PainBow19_5-0\" class=\"reference\"><a href=\"#cite_note-PainBow19-5\">[4]<\/a><\/sup><\/li><\/ul>\n<p>A Google search of \"guidelines for maintaining laboratory notebooks\" or something similar will provide more examples, including those developed by leading universities.\n<\/p><p>At this point, you\u2019re probably wondering why we\u2019re spending so much time on this. The point: good record keeping is the foundation for documenting scientific work regardless of the media, be it paper or electronic. Yes, the laboratory notebook has an electronic equivalent: the <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_laboratory_notebook\" title=\"Electronic laboratory notebook\" class=\"wiki-link\" data-key=\"a9fbbd5e0807980106763fab31f1e72f\">electronic laboratory notebook<\/a> (ELN). These ELNs and other <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_informatics\" title=\"Laboratory informatics\" class=\"wiki-link\" data-key=\"00edfa43edcde538a695f6d429280301\">laboratory informatics<\/a> systems have to support everything paper systems do or they will fail in ensuring the integrity of documented work.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Using_laboratory_records\">Using laboratory records<\/span><\/h2>\n<p>Laboratory records, whether in laboratory notebooks or some other format, can be acted upon in many ways. Laboratory personnel interact with them by:\n<\/p>\n<ul><li> recording observations, results, instrument data output, photos, and charts<\/li>\n<li> describing research processes, goals, and results<\/li>\n<li> ensuring the authenticity of laboratory work<\/li>\n<li> planning and collaborating on experiments<\/li>\n<li> extracting information for reporting<\/li>\n<li> backing up data<\/li>\n<li> querying data<\/li>\n<li> sharing data<\/li>\n<li> publishing data<\/li>\n<li> archiving and retrieving data<\/li>\n<li> securing data<\/li><\/ul>\n<p>Everything on that list can be done with paper records; however, those activities are easier, faster, and less error prone with electronic systems. Paper records aren\u2019t going away anytime soon, for example when needing to record comments and information that may not have been provided for in electronic systems. This is particularly true as a project team expands from one person to more people. However, the need to have shared access to information becomes a limiting factor in productivity when we rely on paper-based systems. Paper-based systems also depend upon the proximity of people working together, something that became problematic during the <a href=\"https:\/\/www.limswiki.org\/index.php\/COVID-19\" title=\"COVID-19\" class=\"mw-redirect wiki-link\" data-key=\"da9bd20c492b2a17074ad66c2fe25652\">COVID-19<\/a> pandemic. Social distancing requirements made sharing paper-based notebook pages more challenging, requiring scanning and emailing. This was perhaps feasible for small amounts of physical materials, but less so for large projects with significant paper-based records.\n<\/p><p>That brings up another important point concerning ownership: whose data is it? When people are handed a notebook, they are told \u201cthis is your notebook, a place to write down your work and observations, and you are responsible for it.\u201d Depending upon how employment or consulting contracts are written, the content that goes into the notebook belongs to whoever is paying for the work. When I worked in a lab, the notebook I used as mine was referenced by others as \u201cyour notebook\u201d (it even had my name on it) even though it wasn\u2019t mine but rather the company\u2019s property. Yet when it was filled, they took possession of it and archived it. This concept of ownership has become a stumbling block in some organizations when they decide to install an ELN or <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management system<\/a> (LIMS), particularly if there are people who have been working there for a long time and have ingrained behaviors. Those people become concerned that someone is going to see their work in an incomplete state before they\u2019ve reviewed and completed it. It\u2019s their work and they don\u2019t want anyone to look at it until it\u2019s done. While the true owners of the work have always had that right, they may not have exercised it, respecting people\u2019s privacy until the work is complete. If you\u2019re considering an informatics system, does it address that concern about ownership?\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Bringing_informatics_into_the_lab\">Bringing informatics into the lab<\/span><\/h2>\n<p>So far this guide has hinted at the implications of adding laboratory informatics systems into the laboratory, but now it's time to discuss it further. Deciding how and when to bring such systems into the lab depends on a number of factors.\n<\/p><p>1. What is the lab's budget? If an informatics implementation can't be properly funded, don\u2019t start that implementation until it can be. The cost of a laptop computer is trivial compared to the total cost of implementation.\n<\/p><p>2. Do we have in-house access to educated and experienced IT support? That staff should understand that laboratory operations are not just another PC- or Microsoft-dominated arena, but rather an environment which has needs for informatics solutions beyond the typical office solutions. For example, laboratory instruments need to be connected to computers, and the resulting data stores should ideally be integrated to make the data more actionable.\n<\/p><p>3. Are laboratory staff ready to use the technologies and take responsibility for the things that go with it? Staff must be trained in more than how to operate an instrument. Can they back up data? Do they understand the security and data privacy risks associated with handling the data?\n<\/p><p>4. Are organizational policies flexible enough to allow practical use of the technology while still keeping security and data privacy risks in mind? The organization should have some sort of network access both internally and externally. Remote access should be possible, particularly given the circumstances surrounding pandemics and the like. A balanced policy on taking an organizational laptop out of the laboratory should be in place. Policies on cameras should also be reasonable, allowing researchers to capture images of samples and specimens for their notebooks. If organizational policies are too restrictive, the technology's usefulness is largely overshadowed.\n<\/p><p>5. What are the lab's space constraints? The size of a lab and the experiments it must conduct can affect the choice of informatics tools.\n<\/p><p>6. What is the nature of the lab's operations? Is it a research lab, service lab, or a combination of both? If you are in a service lab situation, bringing in informatics support as early as possible is essential to your workflow and sanity. You want to minimize having to deal with two separate processes and procedures: the old way we did it (paper-based) and the informatics-supported way.\n<\/p><p>7. Is your lab\u2019s operation governed by external regulatory requirements? If it\u2019s going to be in the future, you may as well start as though it currently is. Note that everything should be validated, regardless of whether or not the lab is subject regulations. Validation isn't done to satisfy regulators but rather to prove that a process works properly. If you don\u2019t have that proof, what\u2019s the point of using that process? Do you really want to trust what a process produces without proof that it works?\n<\/p><p>Most of the points above are easily understood, but let's go into further detail. Let's start by looking at a simple case of you and your project, where you are the sole contributor, without any need for regulatory oversight. Your primary need is to record your planning notes, observations, results, etc. There are tools within the world of computer software to help with this, most notably the word processor. If you have access to one of those, you probably also have access to spreadsheets and other applications that can make your efforts far easier than working with paper. If you search Google for \u201cword processors as lab notebooks\u201d you will find a number of references, including Dr. Martin Engel's guide to using Microsoft OneNote as an ELN<sup id=\"rdp-ebb-cite_ref-EngelBlog15_6-0\" class=\"reference\"><a href=\"#cite_note-EngelBlog15-6\">[5]<\/a><\/sup> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Labforward_GmbH\" title=\"Labforward GmbH\" class=\"wiki-link\" data-key=\"8a9842138692008e4f5896ee023849ab\">Labforward's<\/a> 2020 ELN selection guide.<sup id=\"rdp-ebb-cite_ref-BungersTheElect20_7-0\" class=\"reference\"><a href=\"#cite_note-BungersTheElect20-7\">[6]<\/a><\/sup>\n<\/p><p>However, simply switching from paper to electronic doesn't mean you're done. There's more to consider, like developing backup policies, addressing witness review, connecting to instruments, and addressing the effects of team expansion, including expanding to more comprehensive purpose-built informatics solutions.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Backup_strategy\">Backup strategy<\/span><\/h3>\n<p>You have the electronic documentation tools and the skills to use them, but what else do you need? A backup strategy is imperative. Imagine a scenario where you are using a desktop computer, laptop, or tablet to do your work and it has one copy of the document you\u2019ve been working on for weeks. You press the power button one morning and nothing happens. However, you are not (completely) worried or panicked but rather largely calm because:\n<\/p>\n<ul><li> you have a local backup on removable media (e.g., flash drive, disk), several instances, in fact, that were backed up at least daily, with backups containing everything on the system you were using (you may have a separate backup of your project);<\/li>\n<li> you have a remote backup on your organization's servers (perhaps on a virtual machine);<\/li>\n<li> you have a <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud-based<\/a> backup of at least your project files, and as much of the system files that the cloud storage permits (depending on bandwidth and cost), all secured with two-factor authentication; and<\/li>\n<li> depending on the operating system you are using, you may have built-in backup-and-recover abilities, e.g. as with Mac OS X's \"Time Machine\" functionality.<\/li><\/ul>\n<p>You've done these things because you've asked yourself \"how much is my work worth to me and my organization?\"\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Witness_review_or_sign-off\">Witness review or sign-off<\/span><\/h3>\n<p>The need for a witness review or sign-off can occur for several reasons, including a potential patent filing or proof that the work was done and data recorded properly, on a certain date, in case it is challenged. One of the ramifications is that you have to identify a second person to be that witness (though this would also be the case if you were using a paper notebook).\n<\/p><p>A second issue is that you would have to format the pages of your word processor document (using templates) so as to emulate a signature block and page numbering structure that meets the requirements noted earlier for paper notebooks. You also have to provide a means for either physical (printouts) or <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_signature\" title=\"Electronic signature\" class=\"wiki-link\" data-key=\"dd6997760552a80c6babaf1174c092f4\">electronic signatures<\/a> (e.g., Adobe Acrobat and other applications provide a useful template that could be used as a lab notebook in this case, doing a cut-and-paste from a Word file). You would also have to ensure that once dated and signed, no edits could be made to that material. If you choose the printed route, then you\u2019re back to paper management. One possibility for dealing with that is to scan the signed pages and upload them to a secure server using the file server's date and time stamp system, or to a document management system to demonstrate that documents haven\u2019t been changed.\n<\/p><p>There is another possibility for time-stamping printed material that is scanned with a high-quality scanner. The concept of machine identification code or \"tracking dots\" allows a set of barely perceptible, forensically traceable colored dots to be printed onto pages, with the arrangement of the dots providing information that identifies the printer (by serial number), as well as the date and time a page was printed. Recent research has demonstrated ways to decode these dots for analysis, particularly as part of printer forensics.<sup id=\"rdp-ebb-cite_ref-RichterForensic18_8-0\" class=\"reference\"><a href=\"#cite_note-RichterForensic18-8\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BaraniukWhy17_9-0\" class=\"reference\"><a href=\"#cite_note-BaraniukWhy17-9\">[8]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Instrument-computer_integration\">Instrument-computer integration<\/span><\/h3>\n<p>Using a computer to host a laboratory notebook raises another concern: how can lab instruments or separate instrument data systems connect to automatically transmit data and information to the computer? Controlled data collection will require software beyond a simple word processor, though many of today's ELN vendors provide integration components with their solutions. This connection, possibly using a spreadsheet import, will greatly improve laboratory productivity and the reliability of data transfers.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Expanding_the_research_team\">Expanding the research team<\/span><\/h3>\n<p>Increasing the size of your project by adding more people will have a significant impact on the complexity of using a more electronic laboratory workflow. However, while the basic issues of sharing, collaboration, etc. are not any different than they would be if paper-based notebooks were in use, the electronic solutions to handling those issues are much more useful. At the core of an expanded laboratory operation is getting personnel to agree on how they are going to communicate data and information, how they are going to collaborate using that data and information, and how they will make that agreement sustainable. There is going to be some small allowance for individual approaches to laboratory activities, but critical matters such as data organization and management need to be strictly adhered to. As such, there are several issues to be mindful of.\n<\/p><p>1. How will data and information be organized and compiled? If multiple people are contributing to a project, the results of their work need to be organized in one place so that the status of the work can be ascertained, missing data can be identified, and material is easier to work with. As a project's direction evolves, the formatting and content may change, potentially requiring a complete overhaul of the data structure. That is just a consequence of the way research programs progress.\n<\/p><p>2. How will data and information be updated, versioned, and retained? If your lab works with paper files, this isn't so difficult. The lab may have one printed file detailing experimental methods, and when that method file gets updated, the old printed document is removed and the new one added. In labs where each person has their own individual method file, the process would be repeated for each person. As such, there's no confusion as to the current version; methods would have revision histories so that you would know why they were changed. In cases where those methods are kept in electronic files, more attention has to be paid to ensuring that old method files are archived, and that everyone is working with the same version. This means clear communications procedures are essential. Additionally, the name of the file should have the current revision information; don\u2019t rely on the computer's creation or modification dates for files.\n<\/p><p>3. How is access to common files controlled? Having people edit or add to common files without careful lockout controls is dangerous. If two people open the same document at the same time and make changes, the one who saves the file last can overwrite any changes the other individual has made. You need a document management system that prevents this; if a file is checked out by one person, others may read it but not write to it until the first person is done. Old versions of files should be archived but not deleted so that a detailed revision history is maintained.\n<\/p><p>If an organization grows too large for a consensus-based cooperation of people using office suite products, it will be time to transition to a more purpose-built solution like a multiuser ELN that is capable of allowing multiple people to contribute to a project simultaneously, providing a common organization for data and information, and allowing users to either import data from an instrument or have a direct connection between the ELN and the instrument data system (or through the use of <a href=\"https:\/\/www.limswiki.org\/index.php\/Scientific_data_management_system\" title=\"Scientific data management system\" class=\"wiki-link\" data-key=\"9f38d322b743f578fef487b6f3d7c253\">scientific data management system<\/a> [SDMS]). But how large is too large? It may be the point when personnel become frustrated with waiting for access to documents, or when processes just don\u2019t seem to move as smoothly as they used to. While there is a significant cost factor in implementing an ELN, it should be done sooner rather than later so your lab can benefit from the organizational structure that an ELN provides and reduce the amount of effort it will take to import data and information into the structure. Ideally you are better off if you can start your work with an ELN once the initial project startup work is taken care of. \n<\/p><p>ELNs afford a number of additional benefits, including but not limited to access to:\n<\/p>\n<ul><li> vendor databases for ordering supplies and managing inventory,<\/li>\n<li> chemical structure databases,<\/li>\n<li> reaction databases, and<\/li>\n<li> external libraries.<\/li><\/ul>\n<p>ELNs are appropriate for both basic research and applied research, including work in testing labs where method development and special project work is being done. The prior bulleted list might give you the impression that ELN are oriented toward chemistry; however, that is a reflection of my experience not the industry. The ELN is used in many other industries. Several examples of ELNs past and present, covering many industries, are listed below.\n<\/p>\n<ul><li> LabTech Notebook was released in 1986 and discontinued in 2004, and it was designed to provide communications between computers and laboratory instruments that used RS-232 serial communications. This ELN was applicable to a variety of industries and disciplines.<\/li>\n<li> SmartLab from Velquest was released in the early 2000s and was the first commercial product to carry the \"electronic laboratory notebook\" identifier. It was designed as a platform to encode and guide people conducting lab procedures in GLP\/GMP\/GALP environments. Now owned by <a href=\"https:\/\/www.limswiki.org\/index.php\/Dassault_Syst%C3%A8mes_SA\" title=\"Dassault Syst\u00e8mes SA\" class=\"wiki-link\" data-key=\"1be69bd73e35bc3db0c3229284bf9416\">Dassault Syst\u00e8mes<\/a> and rebranded as a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_execution_system\" title=\"Laboratory execution system\" class=\"wiki-link\" data-key=\"774bdcab852f4d09565f0486bfafc26a\">laboratory execution system<\/a> (LES) as part of the BIOVIA product line, the solution's same conceptual functionality has since been incorporated into LIMS and ELNs that fit the more current expectation for an ELN.<\/li>\n<li> Wolfram Research has as series of notebook products geared toward mathematics.<\/li><\/ul>\n<p>There is a growing list of ELN systems in a variety of disciplines. Given that the scope of activities an ELN can cover is fairly broad, you have to be careful to define your needs before uttering the words \u201cwe\u2019re going to implement an ELN.\u201d We\u2019ll address that point later.\n<\/p><p>The organization of an ELN application is flexible. The layout is user-defined and can contain queryable text, figures, graphics, and fields for instrument-generated data. Because the ELN is inherently flexible and there usually isn\u2019t any quick-start structure, you have to know what you are looking for in a product and how you want to use it. This requires quality due-dilligence research. \u201cIf only I had known\u201d are among the saddest words after product selection.\n<\/p><p>Some organizations have chosen to develop their own ELNs. That is something that should be undertaken with fear and trepidation. Part of the original justification for this route is typically based on the belief that you have special needs that can't be met otherwise. Another concern may be that you don\u2019t want to tie yourself to a vendor that may go out of business. Those points are more a matter of product selection criteria than a justification for a software development project. If you do choose to go with an internal or even a contracted development route, you will potentially have to live with a product that has just one customer (unless the contractor decides to productize it, which is an entirely different discussion). You will also be saddled with the support of that product for the rest of its functional life. And that doesn\u2019t even get into the \"who\" of software product management and development, the \u201cwhen will it be done,\u201d and the inevitable scope creep (i.e., the change and expansions of development requirements). In the initial stages of research projects, the organization of data is subject to change as needs change. This is often at the behest of \u201cunstructured\u201d data and the need to manage it. (Note that it isn\u2019t that the data is truly unstructured, it\u2019s just in a variable structure until the project finds its direction.) This can lead to frustration in setting up a commercial ELN system, let alone designing one from scratch. The next section will look at an organization that is more highly structured.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Meeting_the_needs_of_the_testing_laboratory\">Meeting the needs of the testing laboratory<\/span><\/h2>\n<p>The earlier print shop description will give you an idea of the workflow of a testing or service lab, for example a quality control (QC) lab. In a manually managed QC or research laboratory, operations can become overwhelming quickly. Imagine a study of 40 samples<sup id=\"rdp-ebb-cite_ref-10\" class=\"reference\"><a href=\"#cite_note-10\">[b]<\/a><\/sup> with multiple tests per sample, each of which has to be individually cataloged. Actually I don\u2019t have to imagine it; it happened frequently in our lab, which supported several research groups. When one of these large sample sets shows up\u2014and sometimes more than one\u2014you don't talk to the person doing the sample logging for fear of disrupting their required concentration. You can get them coffee, but no talking.\n<\/p><p>With a LIMS, this isn't so much an issue. You can log in one sample or a hundred, simply by telling it the starting sample ID, how many samples, and what tests should be scheduled. The LIMS then organizes everything and prints the labels for each sample. With some systems, the requestor of a test can even log them in from a web portal, and the central LIMS automatically updates when the samples actually arrive in the lab. \n<\/p><p>A LIMS makes life easier for laboratories in a number of other ways as well. Want to find a list of samples that are pending a particular test? A quality LIMS can readily display that information, including the sample numbers, priorities, and current locations, with no need to manually check work request sheets. Does a third party want to find out the status of one or more of their in-process samples? Role-based access management means a third party can receive limited access to view that status, without seeing anyone else's sensitive data. What about verifying and approving results? The LIMS can provide some level of results checking, with final verification and approval by lab management. When approved, the reports for each set of requests can be printed, emailed, or stored for portal access. And what about integrating data and systems? The LIMS can be connected to an instrument data system (IDS). Depending on the sophistication of that system, the LIMS can generate a worklist of samples that needs to be processed by that device, with the list downloaded to the IDS. When the work is completed, the results can be uploaded directly to the LIMS. This type of system interaction is one of the places where significant productivity gains can be had.\n<\/p><p>The key attributes of a LIMS are shown in Figure 6.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Liscouski_AppInfoSciWork21.png\" class=\"image wiki-link\" data-key=\"d409444eac6274c0b9a34c2bd8e1a551\"><img alt=\"Fig6 Liscouski AppInfoSciWork21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d3\/Fig6_Liscouski_AppInfoSciWork21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> Slide detailing the core components of a LIMS, from <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:A_Guide_for_Management:_Successfully_Applying_Laboratory_Systems_to_Your_Organization%27s_Work\" title=\"LII:A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work\" class=\"wiki-link\" data-key=\"00b300565027cb0518bcb0410d6df360\">A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work<\/a><\/i>, Part 1: Laboratory Informatics Technologies, a webinar series by Joe Liscouski<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>These are just the highlights of what a LIMS can do. Why is a LIMS more effective out-of-the-box than an ELN? The operational behavior of testing labs is essentially the same across industries and disciplines, and as a result vendors have a more predictable and stable base of customers and user requirements to work against than an ELN. It\u2019s a bit like comparing home building contractors. Some have a basic structural architecture that they can modify to meet the needs of a broad market. Others do completely custom homes from the basement up, each of which is unique. The economies of scale and a broader, more predictable customer base show up in products that are easier to work with and adapt both for the primary vendor and those providing add-ons. Those LIMS add-ons include specialized facilities for different industries, including enology and viticulture analysis, water treatment, mineral analysis, cannabis testing, and clinical diagnostics (though the system used in the clinical setting is typically referred to as a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> or LIS). Regardless, and as is the case with ELNs, you want to install a LIMS as soon as you are able to avoid issues surrounding differing workflows based on pre- and post-implementation.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Other_laboratory_informatics_systems\">Other laboratory informatics systems<\/span><\/h2>\n<p>We've mentioned a few other systems in passing, but here we'll provide a brief overview of a few of them.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Scientific_data_management_system_.28SDMS.29\">Scientific data management system (SDMS)<\/span><\/h3>\n<p>The SDMS helps laboratories better solve the problem of dealing with a large number of data files that are being generated, basically by acting as a giant file cabinet that LIMS, ELN, and IDS can connect to. For example, you may not want to put large datasets, spectra, etc. in a LIMS or ELN, you but still have to reference those large files within other internal LIMS and ELN files. This is where the SDMS steps in, storing those large files in the \"file cabinet\" while maintaining references and metadata that are usable by other informatics systems.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Laboratory_execution_system_.28LES.29\">Laboratory execution system (LES)<\/span><\/h3>\n<p>You may not know about them, but there are lesser-known systems called LES that are designed to ensure testing protocols are executed properly and that all the data that is necessary is properly recorded. The initial incarnation arose from the previously discussed SmartLab<sup id=\"rdp-ebb-cite_ref-11\" class=\"reference\"><a href=\"#cite_note-11\">[c]<\/a><\/sup>, a stand-alone product that would guide an analyst through the steps of an analysis, recording each detail of the work in a file that regulatory agencies could inspect to ensure that work was done properly. It found a ready market in any lab that needed to meet GLP\/GMP requirements. The functionality needed to create the same capability can be found in some LIMS and ELNs, but programming is required.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Instrument_data_system_.28IDS.29\">Instrument data system (IDS)<\/span><\/h3>\n<p>Any laboratory instrument popular in the marketplace has had a computer either attached to it or built into it as a package. That\u2019s what an IDS is. It provides automated control over an instrument, collecting and analyzing the data produced by the measuring components. Depending on the sophistication of the vendor, and the demands of the marketplace, the connections between the IDS and another laboratory informatics solution may range from user-programmable interfaces (via a network, USB, serial, or digital I\/O connection) to built-in communications systems that are almost plug-and-play. The latter are most commonly found in the clinical chemistry market, where a great deal of attention has been paid to integration and systems communication via <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"e0bf845fb58d2bae05a846b47629e86f\">Health Level 7<\/a> (HL7) and related protocols. (The details, however, are beyond the scope of this document.)\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Planning_for_laboratory_informatics\">Planning for laboratory informatics<\/span><\/h2>\n<p>There are two key requirements to the successful implementation of informatics products: education and planning. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Education\">Education<\/span><\/h3>\n<p>As far as education goes, the webinar series noted in Figure 6 is a good tool, as are documents provided by technical standards body <a href=\"https:\/\/www.limswiki.org\/index.php\/ASTM_International\" title=\"ASTM International\" class=\"wiki-link\" data-key=\"dfeafbac63fa786e77b472c3f86d07ed\">ASTM International<\/a>. ASTM documents that may be of value to you include:\n<\/p>\n<ul><li> <i>ASTM E1578-18 Standard Guide for Laboratory Informatics<\/i><sup id=\"rdp-ebb-cite_ref-ASTME1578_12-0\" class=\"reference\"><a href=\"#cite_note-ASTME1578-12\">[9]<\/a><\/sup><\/li>\n<li> <i>ASTM E1578-13 Standard Guide for Laboratory Informatics<\/i><sup id=\"rdp-ebb-cite_ref-ASTME1578-13_13-0\" class=\"reference\"><a href=\"#cite_note-ASTME1578-13-13\">[10]<\/a><\/sup> (contains some information not found in 2018 version)<\/li>\n<li> <i>ASTM E1947-98(2014) Standard Specification for Analytical Data Interchange Protocol for Chromatographic Data<\/i><sup id=\"rdp-ebb-cite_ref-ASTME1947_14-0\" class=\"reference\"><a href=\"#cite_note-ASTME1947-14\">[11]<\/a><\/sup><\/li>\n<li> <i>ASTM D8244-21 Standard Guide for Analytical Laboratory Operations Supporting the Cannabis\/Hemp Industry<\/i><sup id=\"rdp-ebb-cite_ref-ASTMD8244_15-0\" class=\"reference\"><a href=\"#cite_note-ASTMD8244-15\">[12]<\/a><\/sup><\/li>\n<li> If you search the ASTM website for \"LIMS\" or \"informatics,\" you'll be surprised by the amount of other material that shows up.<\/li><\/ul>\n<p>Other sources of educational material include:\n<\/p>\n<ul><li> The Tutorial section of LIMSforum.com<sup id=\"rdp-ebb-cite_ref-LIMSforumTutorial_16-0\" class=\"reference\"><a href=\"#cite_note-LIMSforumTutorial-16\">[13]<\/a><\/sup><\/li>\n<li> <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:Laboratory_Information_Systems_Project_Management:_A_Guidebook_for_International_Implementations\" title=\"LII:Laboratory Information Systems Project Management: A Guidebook for International Implementations\" class=\"wiki-link\" data-key=\"de9b2109f3975a703634052af790b2d1\">Laboratory Information Systems Project Management: A Guidebook for International Implementations<\/a><\/i>, by the APHL<\/li>\n<li> <i>Laboratory Informatics Guide 2021<\/i>, by <i>Scientific Computing World<\/i><sup id=\"rdp-ebb-cite_ref-SCWLabInfo21_17-0\" class=\"reference\"><a href=\"#cite_note-SCWLabInfo21-17\">[14]<\/a><\/sup><\/li>\n<li> <i>Lab Manager<\/i> magazine, published by LabX Media Group<sup id=\"rdp-ebb-cite_ref-LabXLabManager_18-0\" class=\"reference\"><a href=\"#cite_note-LabXLabManager-18\">[15]<\/a><\/sup><\/li>\n<li> <i>Computerized Systems in the Modern Laboratory: A Practical Guide<\/i>, by Joe Liscouski<sup id=\"rdp-ebb-cite_ref-LiscouskiComput15_19-0\" class=\"reference\"><a href=\"#cite_note-LiscouskiComput15-19\">[16]<\/a><\/sup><\/li>\n<li> Any vendor with an informatics product will send you a nearly endless stream of material.<\/li><\/ul>\n<p>Ultimately, however, the responsibility for informatics implementation projects doesn't exist solely on the shoulders of laboratory personnel. Everyone connected with a laboratory implementing informatics solutions should have some level of awareness regarding laboratory informatics, including upper management. However, the level of knowledge required may vary slightly depending on the role. For example, laboratory personnel should be fully educated on common laboratory technologies at a minimum. They should also understand why an informatics project is being considered, what the scope of the implementation will be, and what their role will be in the implementation. Upper management should remember to ask laboratory personnel for input on the project, including the topic of product requirements, in order for personnel to feel like they are part of the process, not simply an observer or \"something that's happening to them.\" Finally, laboratory personnel should understand how their jobs are going to change once the implementation is complete. This needs to be addressed very early in project discussions; it is a matter of change, and change scares people, particularly if it affects their job and income. This is not a \u201cwe\u2019ll deal with that later\u201d point. Don\u2019t start the discussion until you figure this out. Things may change, but people want security.\n<\/p><p>Of course, any information technology (IT) personnel will also be involved, requiring significant knowledge about not only networking and software installations but also systems integrations. IT personnel need to understand their role in the implementation project, which can include support, project management, evaluating the vendor support capabilities, and more. They should also be fully aware of and understand the technologies the organization is considering for implementation, and they will be a vital part of the project planning and vendor selection process. IT personnel also will be interested in questions about any <a href=\"https:\/\/www.limswiki.org\/index.php\/Enterprise_resource_planning\" title=\"Enterprise resource planning\" class=\"wiki-link\" data-key=\"07be791b94a208f794e38224f0c0950b\">enterprise resource planning<\/a> (ERP) aspects, which may raise issues of \"build or buy.\" The organization needs to be prepared to both address these concerns and gain IT personnel as strong supporters of the project.\n<\/p><p>Finally, upper management\u2014those who are going to approve the project and provide funding\u2014need to be educated enough to understand the benefits and risks of the proposed implementation, including the likely time scale for the work. Upper management will need to be active in the project in at least two critical junctures, plus at specific milestones as needed. The first time upper management will need informed participation will be during initial project planning. They will help the organization lay out the issues that need to be addressed, the scope of options that will be investigated, and how the organization is going to proceed. They may pose questions such as \u201ccan we use existing system to solve the problem,\u201d particularly if there already has been an investment in an ERP solution such as SAP. Such technology questions will also be of interest to IT personnel since they have an investment in those systems. The second time upper management needs to undoubtedly be involved is when the actual project proposal is finished and is ready to be pitched. They will ask need to ask questions about the reasoning behind the choices made, why current systems are insufficient, what kind of investment the project will require, how the implementation will be scheduled, and how the roll-out would be planned. Understanding the answers to these and other questions will be difficult if upper management doesn\u2019t understand the technology, the issues, the options, and the benefits of the proposed laboratory informatics project.\n<\/p><p>If the world of informatics is new to any or all these stakeholders, the organization must consider getting outside support (not from a vendor) to guide the organization through the process of evaluating the lab's operations, scoping out an overall implementation program, dividing it into milestones, setting priorities, and developing user requirements.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Planning\">Planning<\/span><\/h3>\n<p>The whole point of project planning is to get your organization from the starting point to the end goal, preferably using the most effective path. So, where is the organization going, and why? Everything pretty much boils down to those two questions. Once those questions are answered, more will arise concerning how the organization is actually going to get to the project's end goal.\n<\/p><p>Initial planning will look at the lab from a standpoint of the minimum number of computers required to do the work. In some cases, perhaps just those computational systems built into instruments will be sufficient. Whatever the decision, that\u2019s the planning baseline. Then consider what kind of lab it is: a research lab, a service lab, or a blend? That helps direct concentration on potential solutions; however, be sure not to completely eliminate other options just yet. For example, if your lab is a QC lab, it's probably a service lab in need of a LIMS, but even then there are still options to evaluate.\n<\/p><p>From there, the organization must also think in terms of where that baseline lab is going to be in five years; it\u2019s an arbitrary number but a starting point. Why that far out? It will take a year or two to plan, implement, validate, and educate people to work with the new systems and for the lab to change its behavior and settle down to a new mode of working. The organization should also consider what other changes are likely to take place. What new equipment, procedures, and management changes can be anticipated? Ideally any implemented informatics system will be in place and stable for a few years before people start asking for significant changes; minor ones will likely happen early on. Any hint of \u201cwe didn\u2019t plan for that\u201d will be viewed as poor leadership. Figure 7 shows some of the key points you need to look at during planning.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Liscouski_AppInfoSciWork21.png\" class=\"image wiki-link\" data-key=\"3af5b365b0632c3c57c0acfe6b912fd8\"><img alt=\"Fig7 Liscouski AppInfoSciWork21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/18\/Fig7_Liscouski_AppInfoSciWork21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 7.<\/b> Slide detailing considerations of laboratory informatics implementation in the lab, from <i><a href=\"https:\/\/www.limswiki.org\/index.php\/LII:A_Guide_for_Management:_Successfully_Applying_Laboratory_Systems_to_Your_Organization%27s_Work\" title=\"LII:A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work\" class=\"wiki-link\" data-key=\"00b300565027cb0518bcb0410d6df360\">A Guide for Management: Successfully Applying Laboratory Systems to Your Organization's Work<\/a><\/i>, Part 4: LIMS\/LIS, ELN, SDMS, IT & Education, a webinar series by Joe Liscouski<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Another factor that needs to be considered is that the considerations shown in Figure 7 can be repeated for each lab in the organization. The project planning team also needs to consider how current laboratory workflow impacts other labs. Are there synergistic effects that can be gained by broadening the scope of what the lab is doing?\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Why_projects_fail\">Why projects fail<\/span><\/h3>\n<p>We shouldn't finish the planning section of this guide without discussing why laboratory informatics implementation projects can and do fail. These types of projects are large software projects, and delays, cost over-runs, and failures are common (but hopefully not for your organization's project). Just Google \"why IT projects fail\" and read through some anecdotes. The following are some common reasons informatics implementation projects fail.\n<\/p>\n<ul><li> Insufficient budgeting: Projects can run short of funding, requiring an awkward meeting with management asking for additional funding (without a project change to account for it), inevitably showing a lack of foresight and planning. Build in a large contingency fund because the unexpected happens. If you\u2019d like some education on the topic, watch a few episodes of any home upgrade project on HGTV, in particular <i>Love It or List It<\/i>.<sup id=\"rdp-ebb-cite_ref-20\" class=\"reference\"><a href=\"#cite_note-20\">[d]<\/a><\/sup><\/li>\n<li> Insufficient management support: If sufficient communication isn\u2019t made with management, problems may arise. For example, project delays are a fact of life. Keep clear communications with upper management so that they, and everyone else on the project, know what is going on. Miscommunication or lack of communication of other aspects of the project may inevitably doom the project.<\/li>\n<li> Poor understanding of the scope and nature of the project: This is an educational issue, and a lack of education for all involved parties is almost a guarantee of failure. If you need help, bring in an independent consultant who can lend confidence to the project and its management.<\/li>\n<li> Lack of quality or attention to detail: \u201cThere is always time to do it over, but not enough to do it right\u201d is a common complaint in engineering projects. If you hear it on your organization's project, you are in trouble. Basically the complaint is that project members are cutting corners and not doing things properly, and in a way that is not supportable. This never ends well; sometimes not quickly, but in the long run it leads to problems. <\/li>\n<li> Poor or unrealistic timelines: You may as well face reality from the start: an aggressive timeline just leads to problems (see bullet above). Timelines expand, but they almost never get shorter. If the project team is always rushing, something will most certainly get missed, causing problems later down the road.<\/li>\n<li> Poor project management: Well-managed projects are obvious, and so are poorly managed ones. Just watch the people working, their demeanor, and their attitude about the work; it will tell you all you need to know. Well-managed projects may not always run smoothly, but they make consistent progress. Poorly managed projects cause you to make excuses.<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Closing\">Closing<\/span><\/h2>\n<p>This guide is like the directory signs in shopping malls: they tell you where you are and what shops and restaurants in the facility to consider. Once you figure out what you are looking for, you can find your way there. Hopefully in reading this you\u2019ve formed an idea of what you want to look at and what your path to finding it is.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Footnotes\">Footnotes<\/span><\/h2>\n<div class=\"reflist\" style=\"list-style-type: lower-alpha;\">\n<ol class=\"references\">\n<li id=\"cite_note-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-1\">\u2191<\/a><\/span> <span class=\"reference-text\">According to Greek mythology (from the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/myths.e2bn.org\/mythsandlegends\/userstory14957-why-is-the-sky-blue.html\" target=\"_blank\">E2BN Myths page<\/a>): \"Long long ago, when Queen Athena (Zeus's daughter) was born, Zeus blessed her with two boons for when she came of age. After almost 15 years, Athena was told to think up two things to ask for ... 1) To have a city in Greece named after her (Athens) [and] 2) To have all the people of the world see her face every day of the year (what you are seeing are only her eyes). Thus, the sky is blue, just like the color of Athena's eyes...\"<\/span>\n<\/li>\n<li id=\"cite_note-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-10\">\u2191<\/a><\/span> <span class=\"reference-text\">In some life science drug screening studies, the number can be far higher, which is where robotics and automation becomes important.<\/span>\n<\/li>\n<li id=\"cite_note-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-11\">\u2191<\/a><\/span> <span class=\"reference-text\">A name that has been used by conference organizers after the product was sold and renamed. If you do a Google search on \u201cSmartLab,\u201d you may be surprised at what turns up.<\/span>\n<\/li>\n<li id=\"cite_note-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-20\">\u2191<\/a><\/span> <span class=\"reference-text\">Some season highlights <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.hgtv.com\/shows\/love-it-or-list-it\" target=\"_blank\">can be found<\/a> on the HGTV website.<\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"About_the_author\">About the author<\/span><\/h2>\n<p>Initially educated as a chemist, author Joe Liscouski (joe dot liscouski at gmail dot com) is an experienced laboratory automation\/computing professional with over forty years of experience in the field, including the design and development of automation systems (both custom and commercial systems), LIMS, robotics and data interchange standards. He also consults on the use of computing in laboratory work. He has held symposia on validation and presented technical material and short courses on laboratory automation and computing in the U.S., Europe, and Japan. He has worked\/consulted in pharmaceutical, biotech, polymer, medical, and government laboratories. His current work centers on working with companies to establish planning programs for lab systems, developing effective support groups, and helping people with the application of automation and information technologies in research and quality control environments.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-SBPDuplicate-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SBPDuplicate_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/scientificbindery.com\/products\/duplicate-lined-notebook\/\" target=\"_blank\">\"Duplicate Lined Notebook\"<\/a>. Scientific Bindery Productions, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/scientificbindery.com\/products\/duplicate-lined-notebook\/\" target=\"_blank\">https:\/\/scientificbindery.com\/products\/duplicate-lined-notebook\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Duplicate+Lined+Notebook&rft.atitle=&rft.pub=Scientific+Bindery+Productions%2C+Inc&rft_id=https%3A%2F%2Fscientificbindery.com%2Fproducts%2Fduplicate-lined-notebook%2F&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WestGMP-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WestGMP_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">West, D. (n.d.). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/users.stlcc.edu\/departments\/fvbio\/Lab_Practices_GLP_STLCC.htm\" target=\"_blank\">\"GMP\/GLP Recordkeeping\"<\/a>. St. Louis Community College, Center for Plant and Life Sciences<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/users.stlcc.edu\/departments\/fvbio\/Lab_Practices_GLP_STLCC.htm\" target=\"_blank\">https:\/\/users.stlcc.edu\/departments\/fvbio\/Lab_Practices_GLP_STLCC.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GMP%2FGLP+Recordkeeping&rft.atitle=&rft.aulast=West%2C+D.&rft.au=West%2C+D.&rft.date=n.d.&rft.pub=St.+Louis+Community+College%2C+Center+for+Plant+and+Life+Sciences&rft_id=https%3A%2F%2Fusers.stlcc.edu%2Fdepartments%2Ffvbio%2FLab_Practices_GLP_STLCC.htm&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NIHGuidelines08-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NIHGuidelines08_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Institutes of Health (December 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/oir.nih.gov\/sites\/default\/files\/uploads\/sourcebook\/documents\/ethical_conduct\/guidelines-scientific_recordkeeping.pdf\" target=\"_blank\">\"Guidelines for Scientific Record Keeping in the Intramural Research Program at the NIH\"<\/a> (PDF). Office of the Director<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/oir.nih.gov\/sites\/default\/files\/uploads\/sourcebook\/documents\/ethical_conduct\/guidelines-scientific_recordkeeping.pdf\" target=\"_blank\">https:\/\/oir.nih.gov\/sites\/default\/files\/uploads\/sourcebook\/documents\/ethical_conduct\/guidelines-scientific_recordkeeping.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Guidelines+for+Scientific+Record+Keeping+in+the+Intramural+Research+Program+at+the+NIH&rft.atitle=&rft.aulast=National+Institutes+of+Health&rft.au=National+Institutes+of+Health&rft.date=December+2008&rft.pub=Office+of+the+Director&rft_id=https%3A%2F%2Foir.nih.gov%2Fsites%2Fdefault%2Ffiles%2Fuploads%2Fsourcebook%2Fdocuments%2Fethical_conduct%2Fguidelines-scientific_recordkeeping.pdf&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PainBow19-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PainBow19_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pain, E. (03 September 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sciencemag.org\/careers\/2019\/09\/how-keep-lab-notebook\" target=\"_blank\">\"How to keep a lab notebook\"<\/a>. <i>Science<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.caredit.aaz3678\" target=\"_blank\">10.1126\/science.caredit.aaz3678<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sciencemag.org\/careers\/2019\/09\/how-keep-lab-notebook\" target=\"_blank\">https:\/\/www.sciencemag.org\/careers\/2019\/09\/how-keep-lab-notebook<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+to+keep+a+lab+notebook&rft.atitle=Science&rft.aulast=Pain%2C+E.&rft.au=Pain%2C+E.&rft.date=03+September+2019&rft_id=info:doi\/10.1126%2Fscience.caredit.aaz3678&rft_id=https%3A%2F%2Fwww.sciencemag.org%2Fcareers%2F2019%2F09%2Fhow-keep-lab-notebook&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EngelBlog15-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EngelBlog15_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Engel, M. (December 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/martinengel.net\/2015\/12\/how-to-use-onenote-as-your-electronic-notebook\/\" target=\"_blank\">\"Blog: How to use onenote as your electronic lab book\"<\/a>. <i>MartinEngel.net<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/martinengel.net\/2015\/12\/how-to-use-onenote-as-your-electronic-notebook\/\" target=\"_blank\">http:\/\/martinengel.net\/2015\/12\/how-to-use-onenote-as-your-electronic-notebook\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Blog%3A+How+to+use+onenote+as+your+electronic+lab+book&rft.atitle=MartinEngel.net&rft.aulast=Engel%2C+M.&rft.au=Engel%2C+M.&rft.date=December+2015&rft_id=http%3A%2F%2Fmartinengel.net%2F2015%2F12%2Fhow-to-use-onenote-as-your-electronic-notebook%2F&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BungersTheElect20-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BungersTheElect20_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bungers, S. (2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labfolder.com\/electronic-lab-notebook-eln-research-guide\" target=\"_blank\">\"The Electronic Lab Notebook in 2020: A comprehensive guide\"<\/a>. Labforward GmbH<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.labfolder.com\/electronic-lab-notebook-eln-research-guide\" target=\"_blank\">https:\/\/www.labfolder.com\/electronic-lab-notebook-eln-research-guide<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Electronic+Lab+Notebook+in+2020%3A+A+comprehensive+guide&rft.atitle=&rft.aulast=Bungers%2C+S.&rft.au=Bungers%2C+S.&rft.date=2020&rft.pub=Labforward+GmbH&rft_id=https%3A%2F%2Fwww.labfolder.com%2Felectronic-lab-notebook-eln-research-guide&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RichterForensic18-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RichterForensic18_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Richter, T.; Escher, S.; Sch\u00f6nfeld, D. et al. (2018). \"Forensic Analysis and Anonymisation of Printed Documents\". <i>Proceedings of the 6th ACM Workshop on Information Hiding and Multimedia Security<\/i>: 127\u201338. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3206004.3206019\" target=\"_blank\">10.1145\/3206004.3206019<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Forensic+Analysis+and+Anonymisation+of+Printed+Documents&rft.jtitle=Proceedings+of+the+6th+ACM+Workshop+on+Information+Hiding+and+Multimedia+Security&rft.aulast=Richter%2C+T.%3B+Escher%2C+S.%3B+Sch%C3%B6nfeld%2C+D.+et+al.&rft.au=Richter%2C+T.%3B+Escher%2C+S.%3B+Sch%C3%B6nfeld%2C+D.+et+al.&rft.date=2018&rft.pages=127%E2%80%9338&rft_id=info:doi\/10.1145%2F3206004.3206019&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaraniukWhy17-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BaraniukWhy17_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Baraniuk, C. (07 June 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20191102031255\/https:\/\/www.bbc.com\/future\/article\/20170607-why-printers-add-secret-tracking-dots\" target=\"_blank\">\"Why printers add secret tracking dots\"<\/a>. <i>BBC Future<\/i>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bbc.com\/future\/article\/20170607-why-printers-add-secret-tracking-dots\" target=\"_blank\">the original<\/a> on 02 November 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20191102031255\/https:\/\/www.bbc.com\/future\/article\/20170607-why-printers-add-secret-tracking-dots\" target=\"_blank\">https:\/\/web.archive.org\/web\/20191102031255\/https:\/\/www.bbc.com\/future\/article\/20170607-why-printers-add-secret-tracking-dots<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Why+printers+add+secret+tracking+dots&rft.atitle=BBC+Future&rft.aulast=Baraniuk%2C+C.&rft.au=Baraniuk%2C+C.&rft.date=07+June+2017&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20191102031255%2Fhttps%3A%2F%2Fwww.bbc.com%2Ffuture%2Farticle%2F20170607-why-printers-add-secret-tracking-dots&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ASTME1578-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ASTME1578_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.astm.org\/Standards\/E1578.htm\" target=\"_blank\">\"ASTM E1578-18 Standard Guide for Laboratory Informatics\"<\/a>. ASTM International. 2018. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1520%2FE1578-18\" target=\"_blank\">10.1520\/E1578-18<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.astm.org\/Standards\/E1578.htm\" target=\"_blank\">https:\/\/www.astm.org\/Standards\/E1578.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ASTM+E1578-18+Standard+Guide+for+Laboratory+Informatics&rft.atitle=&rft.date=2018&rft.pub=ASTM+International&rft_id=info:doi\/10.1520%2FE1578-18&rft_id=https%3A%2F%2Fwww.astm.org%2FStandards%2FE1578.htm&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ASTME1578-13-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ASTME1578-13_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.astm.org\/DATABASE.CART\/HISTORICAL\/E1578-13.htm\" target=\"_blank\">\"ASTM E1578-13 Standard Guide for Laboratory Informatics\"<\/a>. ASTM International. 2013. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1520%2FE1578-13\" target=\"_blank\">10.1520\/E1578-13<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.astm.org\/DATABASE.CART\/HISTORICAL\/E1578-13.htm\" target=\"_blank\">https:\/\/www.astm.org\/DATABASE.CART\/HISTORICAL\/E1578-13.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ASTM+E1578-13+Standard+Guide+for+Laboratory+Informatics&rft.atitle=&rft.date=2013&rft.pub=ASTM+International&rft_id=info:doi\/10.1520%2FE1578-13&rft_id=https%3A%2F%2Fwww.astm.org%2FDATABASE.CART%2FHISTORICAL%2FE1578-13.htm&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ASTME1947-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ASTME1947_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.astm.org\/Standards\/E1947.htm\" target=\"_blank\">\"ASTM E1947-98(2014) Standard Specification for Analytical Data Interchange Protocol for Chromatographic Data\"<\/a>. ASTM International. 2014. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1520%2FE1947-98R14\" target=\"_blank\">10.1520\/E1947-98R14<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.astm.org\/Standards\/E1947.htm\" target=\"_blank\">https:\/\/www.astm.org\/Standards\/E1947.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ASTM+E1947-98%282014%29+Standard+Specification+for+Analytical+Data+Interchange+Protocol+for+Chromatographic+Data&rft.atitle=&rft.date=2014&rft.pub=ASTM+International&rft_id=info:doi\/10.1520%2FE1947-98R14&rft_id=https%3A%2F%2Fwww.astm.org%2FStandards%2FE1947.htm&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ASTMD8244-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ASTMD8244_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.astm.org\/Standards\/D8244.htm\" target=\"_blank\">\"ASTM D8244-21 Standard Guide for Analytical Laboratory Operations Supporting the Cannabis\/Hemp Industry\"<\/a>. ASTM International. 2021. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1520%2FD8244-21\" target=\"_blank\">10.1520\/D8244-21<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.astm.org\/Standards\/D8244.htm\" target=\"_blank\">https:\/\/www.astm.org\/Standards\/D8244.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ASTM+D8244-21+Standard+Guide+for+Analytical+Laboratory+Operations+Supporting+the+Cannabis%2FHemp+Industry&rft.atitle=&rft.date=2021&rft.pub=ASTM+International&rft_id=info:doi\/10.1520%2FD8244-21&rft_id=https%3A%2F%2Fwww.astm.org%2FStandards%2FD8244.htm&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LIMSforumTutorial-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LIMSforumTutorial_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.limsforum.com\/category\/education\/tutorials-education\/\" target=\"_blank\">\"Tutorials\"<\/a>. <i>LIMSforum<\/i>. LabLynx, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.limsforum.com\/category\/education\/tutorials-education\/\" target=\"_blank\">https:\/\/www.limsforum.com\/category\/education\/tutorials-education\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Tutorials&rft.atitle=LIMSforum&rft.pub=LabLynx%2C+Inc&rft_id=https%3A%2F%2Fwww.limsforum.com%2Fcategory%2Feducation%2Ftutorials-education%2F&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SCWLabInfo21-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SCWLabInfo21_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.scientific-computing.com\/issue\/laboratory-informatics-guide-2021\" target=\"_blank\">\"Laboratory Informatics Guide 2021\"<\/a>. Scientific Computing World. 2021<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.scientific-computing.com\/issue\/laboratory-informatics-guide-2021\" target=\"_blank\">https:\/\/www.scientific-computing.com\/issue\/laboratory-informatics-guide-2021<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Laboratory+Informatics+Guide+2021&rft.atitle=&rft.date=2021&rft.pub=Scientific+Computing+World&rft_id=https%3A%2F%2Fwww.scientific-computing.com%2Fissue%2Flaboratory-informatics-guide-2021&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LabXLabManager-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LabXLabManager_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labmanager.com\/magazine\" target=\"_blank\">\"Lab Manager\"<\/a>. LabX Media Group. 2021<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.labmanager.com\/magazine\" target=\"_blank\">https:\/\/www.labmanager.com\/magazine<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Lab+Manager&rft.atitle=&rft.date=2021&rft.pub=LabX+Media+Group&rft_id=https%3A%2F%2Fwww.labmanager.com%2Fmagazine&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiscouskiComput15-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiscouskiComput15_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Liscouski, J.G. (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.dhibooks.com\/computerized-systems-in-the-modern-laboratory-a-practical-guide\" target=\"_blank\"><i>Computerized Systems in the Modern Laboratory: A Practical Guide<\/i><\/a>. DHI Publishing. pp. 432. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781933722863<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.dhibooks.com\/computerized-systems-in-the-modern-laboratory-a-practical-guide\" target=\"_blank\">https:\/\/www.dhibooks.com\/computerized-systems-in-the-modern-laboratory-a-practical-guide<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Computerized+Systems+in+the+Modern+Laboratory%3A+A+Practical+Guide&rft.aulast=Liscouski%2C+J.G.&rft.au=Liscouski%2C+J.G.&rft.date=2015&rft.pages=pp.%26nbsp%3B432&rft.pub=DHI+Publishing&rft.isbn=9781933722863&rft_id=https%3A%2F%2Fwww.dhibooks.com%2Fcomputerized-systems-in-the-modern-laboratory-a-practical-guide&rfr_id=info:sid\/en.wikipedia.org:LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n\n<!-- \nNewPP limit report\nCached time: 20210705150626\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.416 seconds\nReal time usage: 1.946 seconds\nPreprocessor visited node count: 10136\/1000000\nPreprocessor generated node count: 21926\/1000000\nPost\u2010expand include size: 69807\/2097152 bytes\nTemplate argument size: 26757\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 233.346 1 - -total\n 89.57% 209.014 2 - Template:Reflist\n 71.46% 166.753 16 - Template:Citation\/core\n 65.16% 152.045 14 - Template:Cite_web\n 5.85% 13.651 1 - Template:Cite_book\n 5.84% 13.618 1 - Template:Cite_journal\n 4.76% 11.111 4 - Template:Efn\n 4.34% 10.127 21 - Template:Citation\/make_link\n 4.32% 10.087 7 - Template:Citation\/identifier\n 1.62% 3.782 14 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12527-0!*!0!!en!5!* and timestamp 20210705150624 and revision id 42703\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies\">https:\/\/www.limswiki.org\/index.php\/LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","d8b467af534a70312a21f63b61be26cd_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/5c\/Fig1_Liscouski_AppInfoSciWork21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/14\/Fig2_Liscouski_AppInfoSciWork21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/62\/Fig3_Liscouski_AppInfoSciWork21.png","https:\/\/upload.wikimedia.org\/wikipedia\/commons\/0\/0c\/AGBell_Notebook.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/65\/Fig5_Liscouski_AppInfoSciWork21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d3\/Fig6_Liscouski_AppInfoSciWork21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/18\/Fig7_Liscouski_AppInfoSciWork21.png"],"d8b467af534a70312a21f63b61be26cd_timestamp":1625497583,"31bea4f4ab2f82a24f0d0dad004bfbb0_type":"article","31bea4f4ab2f82a24f0d0dad004bfbb0_title":"Using interactive digital notebooks for bioscience and informatics education (Davies et al. 2020)","31bea4f4ab2f82a24f0d0dad004bfbb0_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education","31bea4f4ab2f82a24f0d0dad004bfbb0_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Using interactive digital notebooks for bioscience and informatics education\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nUsing interactive digital notebooks for bioscience and informatics educationJournal\n \nPLOS Computational BiologyAuthor(s)\n \nDavies, Alan; Hooley, Frances; Causey-Freeman, Peter; Eleftheriou, Iliada; Moulton, GeorginaAuthor affiliation(s)\n \nUniversity of ManchesterPrimary contact\n \nEmail: alan dot davies-2 at manchester dot ac dot ukEditors\n \nOuellette, FrancisYear published\n \n2020Volume and issue\n \n16(11)Article #\n \ne1008326DOI\n \n10.1371\/journal.pcbi.1008326ISSN\n \n1553-734XDistribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/journals.plos.org\/ploscompbiol\/article?id=10.1371\/journal.pcbi.1008326Download\n \nhttps:\/\/journals.plos.org\/ploscompbiol\/article\/file?id=10.1371\/journal.pcbi.1008326 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 What is a Jupyter Notebook? \n4 Anatomy of a notebook \n\n4.1 Cell types \n4.2 Styling cells \n\n\n5 Running Jupyter Notebooks \n6 Jupyter Notebooks, JupyterLab, and JupyterHub \n7 Notebook extensions \n8 Magic commands \n9 Widgets \n10 How Jupyter enhances collaboration and reproducibility \n11 Notebooks and assessment \n12 Sharing your notebooks \n13 Case studies \n\n13.1 Case study 1: Modern Information Engineering (MIE) \n13.2 Case study 2: Introduction to Programming \n13.3 Evaluation \n\n\n14 Discussion \n15 Conclusion \n16 Acknowledgements \n\n16.1 Funding \n16.2 Competing interests \n\n\n17 References \n18 Notes \n\n\n\nAbstract \nInteractive digital notebooks provide an opportunity for researchers and educators to carry out data analysis and report results in a single digital format. Further to just being digital, the format allows for rich content to be created in order to interact with the code and data contained in such a notebook to form an educational narrative. This primer introduces some of the fundamental aspects involved in using Jupyter Notebook in an educational setting for teaching in the bioinformatics and health informatics disciplines. We also provide two case studies that detail 1. how we used Jupyter Notebooks to teach non-coders programming skills on a blended master\u2019s degree module for a health informatics program, and 2. a fully online distance learning unit on programming for a postgraduate certificate (PG Cert) in clinical bioinformatics, with a more technical audience.\nKeywords: bioinformatics, health informatics, programming, data analysis, Jupyter Notebook, education\n\nIntroduction \nUniversities and other higher education institutions are now under increasing pressure to provide more online and distance learning courses and to deliver them cost effectively and rapidly.[1] This increase in demand is partly based on students wanting more flexible study options in comparison to traditional higher education course delivery to aid in study around employment and family commitments. This is also driven by financial considerations that allow higher education institutions to scale course delivery while managing infrastructural provision (e.g., access to rooms for teaching and limited capacity for face-to-face delivery).[2] To meet this challenge, we require tools that cater for students with varying levels of digital literacy and reduce the burden of them having to download and install software, all of which requires support, which is more difficult to provide at a distance. This can be further complicated when students use managed equipment (e.g., National Health Service [NHS] employees) and may not have administrator rights to install software.\nDigital notebooks provided us with a way of meeting these needs, as they are easy to set up, straightforward to use, and can support rich and interactive content. Here, we present a primer on how to use digital notebooks (specifically Jupyter Notebooks) for teaching and assessment, along with details of two case studies where we used notebooks to teach Python programming and database skills for clinical bioinformatics and health informatics students of varying levels of technical experience. The case studies and methods presented can be applied to both distance learning and face-to-face teaching scenarios.\nWe will start by covering what a Jupyter Notebook is along with the different \u201ccell\u201d types available. We then look at how they can be run and enhanced with extensions to add items like exercise tasks and other interactivity before looking at how they can be used in assessment. Next, we present two case studies where we have applied notebooks to teach different groups of students to give some examples of the different contexts they can be used in. Finally, we end with a discussion to synthesise our experiences of using notebooks to educate students and their further potential, with considerations for education.\n\nWhat is a Jupyter Notebook? \nJupyter Notebook is an open-source web application that runs in an internet browser. It allows the sharing of code, data analysis, data visualizations (which can be interactive), math formulas, and other embedded media (e.g., YouTube videos, images, and web links), all in a single document combining interactive and narrative components. This takes the form of a document that is composed of multiple cells that encapsulate the content of the notebook (Figure 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 A new Python 3 notebook with three empty cells denoted by the grey rectangles. The currently selected cell is highlighted in green.\n\n\n\nJupyter notebooks were created by Project Jupyter, which, according to their website, states that \u201cProject Jupyter exists to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages.\u201d[3] This includes various standards for interactive computing, including the notebook document format that is based on JavaScript Object Notation (JSON). The name Jupyter is composed of the initial three languages supported: Julia, Python, and R.[4]\n\nAnatomy of a notebook \nJupyter notebooks are available in various programming languages, with current support for over 40 different programming languages.[3] These include the popular languages used for data science, such as Julia, Python, and R (Figure 2).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2 A simple function that returns the value of the sum of two numbers showing different kernels (programming languages) in the notebooks; this example shows Python (left), Julia (middle), and R (right).\n\n\n\nThe notebooks are made up of units called \u201ccells\u201d that can be executed (run) in order to render their contents in different ways.\n\nCell types \nThere are two principle cell types. The first cell type is the \u201cMarkdown\u201d cell, which is used to present text, images, equations, and other resources. The second cell type is the \u201ccode\u201d cell, which allows the user to enter code written in a chosen programming language that will execute in the notebook. To execute the contents of any cell, the user can press the Shift and Enter keys together, or alternatively click on the \u201cRun\u201d button in the main menu bar across the top of the screen. If the cell being run is a code cell, it will cause the code in the cell to be executed and any output displayed immediately below it. This is indicated by the \u201cIn\u201d and \u201cOut\u201d words located to the left of the cells, as seen in Fig 2.\n\nStyling cells \nMarkdown cells can be styled with Markdown, which is a lightweight mark-up language for styling text.[5] This works by turning Markdown text into HTML (Figure 3).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 3 Example of a markdown cell (left) and the output of the styled cell when the cell is run (right).\n\n\n\nThese cells can also display plain text as output with no styling. Another useful feature for teaching math-based courses or sharing formulas, etc. is the integration of LaTeX support. LaTeX is a popular typesetting document preparation system [6] that was built on the Tex typesetting language, originally developed by the American computer scientist Donald Knuth.[6] LaTeX is widely used by the scientific community (e.g., computer scientists) to write academic publications (journal and conference papers). LaTeX math notation can be added to markdown cells to display formulas using common math notation. For example, the code below produces the output seen in Figure 4:\n$\u2009$\n\\sigma = \\sqrt{\\frac{1}{N}\\sum_{i = 1}^{N} (x_i-\\mu)^2}\n$\u2009$<\/tt>\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 4 Output of LaTeX math notation producing the formula for the population standard deviation.\n\n\n\nThe LaTeX wikibook math section[7] is a useful resource for learning about the math notation options available in LaTeX. Table 1 provides an overview of some of the useful Python libraries for numerical and scientific computing that can be incorporated into the notebook environment.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 1. Some useful Python libraries for numerical and scientific computing.\n\n\n\nRunning Jupyter Notebooks \nThere are different ways of accessing Jupyter Notebooks. The Anaconda Distribution[8], a data science platform for Python and R, provides a free Python distribution, which includes Jupyter Notebooks. Another option includes JupyterHub[9], which is designed for groups of users to access notebooks on the cloud or locally hosted and maintained on their own devices. Once run, the user is greeted with a page showing the various files and folders available (Figure 5). Selecting the \u201cnew\u201d option from the menu allows the user to create a new notebook in the selected language; alternatively, an existing notebook (ipynb) file can be loaded by selecting the required file from the list of files in the main list to the left of the screen.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 5 The files and folders tab seen when launching Jupyter notebooks locally. A new notebook is created by selecting the new dropdown option and choosing the required language.\n\n\n\nJupyter Notebooks, JupyterLab, and JupyterHub \nProject Jupyter has created several resources and services surrounding the initial notebooks. This can sometimes cause some confusion among beginners. The difference among them are briefly described here.\n\n Jupyter Notebook is an interactive computational web application that combines code, text, data analysis, and other media in a single document.\n JupyterLab builds on the original Jupyter Notebook to provide an online interactive development environment that allows users to access notebooks with data and file viewers, text editors, and terminals all in the same environment. This helps to better integrate notebooks with other documents and resources in a single environment.\n JupyterHub allows multiple users (groups) to access notebooks and other resources. This can be useful for students and companies that want one or more groups to access and use a computational environment and resources without having to install and set things up. The management of these groups can be carried out by system administrators. Individual notebooks and the JupyterLab can be accessed via the Hub. The Hub can be run in the cloud or on a group's own hardware.\nAs these offerings build on the initial notebook and have notebooks at their core, this article describes the notebooks for beginners, rather than the additional platforms and services that incorporate them. Notebooks themselves work in a similar way regardless of being accessed alone or via JupyterLab or JupyterHub. It is worth being aware of these options, however, for building and sharing resources around the notebooks that you may develop.\n\nNotebook extensions \nA number of different \u201cbolt on\u201d extensions exists for the notebooks. These can be extremely useful for including additional features into a notebook. Some examples include the ability to split a cell into two different cells horizontally, a spellchecker, auto-numbering of equations, and an extension for making exercise tasks (discussed later). To enable and utilize the additional features that are available with the notebooks, the following commands should be entered into the command prompt (e.g., the Anaconda prompt or Powershell):\npip install jupyter_contrib_nbextensions\njupyter contrib nbextension install\u2014user\npip install jupyter_nbextensions_configurator\njupyter nbextensions_configurator enable\u2013user<\/tt>\nThis enables the \u201cNBextensions\u201d tab (Figure 6). When clicked on, the user is presented with a series of checkboxes for the various extensions. There is also a description, often with associated screenshots and\/or animations previewing what the extension does.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 6 The NBextensions tab for selecting the various notebook extensions.\n\n\n\nWhen a new notebook is opened, the selected extensions appear as small icon buttons under the main menu (Figure 7).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 7 Enabled notebook extension icons shown in red box.\n\n\n\nMagic commands \nIPython (the \u201cInteractive Python\u201d kernel used in Jupyter Notebook) also supports what are known as magic commands or functions, which are used to change the standard behaviour of IPython. Magic commands come in two different types: \u201cline\u201d and \u201ccell\u201d magics. The %lsmagic command displays a list of all the available line magics, while %magic displays a help window with information about magic functions. A line magic only works on the line of code that it precedes, whereas a cell magic applies the function to the entire cell. A line magic is prefixed with a single percentage character (%), whereas a cell magic is prefixed with two percent characters (%%). Figure 8 shows an example of this, where we use the magic functions to load a Structured Query Language (SQL) extension and specify a database engine such as SQLite. The second code cell employs cell magic to allow us to write and execute SQL commands in the notebook environment to create a database table. \n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 8 Line and cell magic\u2019s used to add SQL (Structured Query Language) functionality to a Python notebook.\n\n\n\nWidgets \nWidgets can be used for interactive elements in notebooks.[10] Figure 9 shows an example of this where the \u201cinteract\u201d function runs the \u201cget_val\u201d function displaying a slider with the default value (5 in this case) selected. The user can then change the value by moving the slider to the left or right. Figure 10 shows another example, this time using a drop-down list of options created from a Python list.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 9 Example of notebook interaction.\n\n\n\n\n\n\n\n\n\n\n\n\n Fig. 10 An interactive drop-down list created using a Python list.\n\n\n\nA more substantive example of using interactive widgets is highlighted by Richardson and Behrang, who use Python notebooks to view Digital Imaging and Communications in Medicine (DICOM) images.[4]\n\nHow Jupyter enhances collaboration and reproducibility \nReproducibility in science is an important concept. Without reproducability, there is a lack of transparency about what was done. One would expect that if scientists follow the same method, the results will be the same. This is sometimes difficult to achieve with complex data and analysis methods. The quality of research in relation to collaboration was brought into question in a recent Wellcome Trust report on research culture that raised concerns over the impact of lack of research collaboration on research quality, and in some cases, unhealthy competition between researchers.[11] As Hardwicke and colleagues highlight, the availability of data is essential for a self-correcting ecosystem in science, which can be undermined by unclear analysis and poorly curated data, which, in turn, impedes analytic reproducibility.[12]\nThere has been a counter-movement to improve these issues by organizations such as the U.K. Reproducibility Network (UKRN)[13], which is a network of 10 universities in the U.K. that are concerned with reproducibility in research. The founder of UKRN calls for institutional changes to promote open-research practices.[14] Although various research studies do share their data, other researchers\u2019 understanding of the shared dataset and their ability to repeat the previous analysis hinges on the documentation of both the dataset and analysis steps followed, as well as being able to replicate the software environment in order to run the code in the first instance. Because of these requirements, notebooks are being used increasingly by researchers to share analysis code along with an explanation and steps involved in processing the data for reproducible research purposes. This has led to widescale use in the research community.[15] By using interactive notebooks, the data analysis code and steps taken can be shared together with any additional documentation, formulas, etc. that are required to understand the applied method. Sharing data and analysis code in such a way dramatically improves the speed in which the analysis can be rerun by other researchers. Researches are also building on notebook technology for novel purposes, for example, Tellurium notebooks that were developed to support the creation of reproducible models for systems and synthetic biology.[16]\nAside from the research applications, Jupyter Notebooks are also being increasingly used to teach subjects like data science and programming[17], as they feature dynamic responses such as interactive visualizations and rapid updating of results based on the filtering of data (e.g., Figure 11).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 11 Interactive plot generated with the \u201cplotly\u201d module that can be rotated and zoomed with individual data points selected.\n\n\n\nNotebooks and assessment \nNotebooks can also be set up to carry out formative or summative assessment. The \u201cnbgrader\u201d tool[18] allows for the creation and grading of assignments in the notebook environment. The tool allows a user to generate an instructor version of a notebook that has predefined solutions. This, in turn, is used to generate the student version of the notebooks without the solutions. These student versions of the notebook(s) can then be distributed to the students by email or via a virtual learning environment (VLE). The principal aims of the tool were to address issues surrounding the maintenance of separate student and instructor notebook versions, automatic grading of exercises, the manual grading of \u201cfree response\u201d questions, and the ability to provide feedback to students. There are two ways of using the nbgrader. The first is a standalone version, while the second is designed to work with JupyterHub, which can manage the release and collection of submitted assessments. The nbgrader adds a tool bar to each cell to make the cell either an \u201canswer\u201d or a \u201ctest\u201d cell. The answer cells allow students to add code placed between a placeholder. Unit tests are written by the instructor to evaluate the correctness of a student\u2019s solution. Tests can also be hidden from the students. Points can be assigned to each cell to assign specified marks if the unit tests pass. Cells can also be set to \u201cmanually graded\u201d answers so students can write free text, code formulas, etc. Student feedback can be provided when grading by adding text to any required cell and then converting the notebooks into HTML format so they can be emailed or added to VLE for the students to view.\nA simpler method of providing interactive tasks for formative assessment that does not require the knowledge of writing unit tests is to use the exercise extension. This extension can be used to add exercises (Figure 12). Adding feedback in the form of exercises is a unique feature of notebooks that elevates them from simply being an online textbook. The ability to provide interactive tasks that let students engage directly with the notebook without the need to use additional software is a powerful feature. Moreover, this helps maintain the narrative flow, as the exercises can be woven into the content in appropriate places without diverting the user to other tools or resources, all of which helps with the overall user experience.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 12 Example using the \u201cexercise2\u201d extension to create a task. When the \u201cshow solution\u201d button is pressed, the answer is displayed below.\n\n\n\nHere, we give an example of an exercise where we create a task cell and put the solution in the preceding cell. The solution cell can be hidden until the \u201cshow solution\u201d button (Figure 12) is activated, which reveals the hidden cell. This is a good way of adding coding tasks for students and then presenting them with a model answer or solution for comparison and\/or further explanation. Figure 12 shows a task where the student has to add a textual value to a Python dictionary data structure and output the result. Students can attempt to write the code for this and then toggle the solution to check their answer with the one provided.\n\nSharing your notebooks \nNotebooks can be shared in the same way as any other file. In order to run a notebook, however, users will need to install and set up software (i.e., Anaconda). This may not be the ideal solution given that novices may have difficulties installing and setting up the environment required to view and use notebooks. This is further compounded if the user needs to install extra libraries and extensions that may be required to run a notebook as intended. One helpful way around this when sharing notebooks with novices is the Binder project.[19]\nBinder is a web service (currently open-source) that allows users to create interactive sharable and reproducible computational environments in the cloud.[20] Binder uses several different technologies (i.e., repo2docker, JupyterHub, and BinderHub) that allow a user to place their notebooks in a repository (e.g., GitHub). Once done, a form can be filled in on the Binder website (mybinder.org). This includes a repository Uniform Resource Locator (URL), Git tag, and optional path of the notebook file. Following this, a user will receive a URL that they can send to others to share their notebooks.\nFor a more technical explanation of how Binder works, please see the Binder paper presented at the SCIPY conference in 2018 and its associated YouTube video.[20] For more information on how to implement sharing Notebooks with Binder, see the Data Carpentry tutorial, which guides users through sharing their notebooks with GitHub and using Binder.[21]\n\nCase studies \nWe present two short case studies detailing how we have used Jupyter Notebooks to teach programming skills to different audiences on two of our courses, a master's module on a health informatics program and an introduction to programming module on postgraduate certificate (PG Cert) in clinical bioinformatics. This is followed by a brief initial evaluation of the use of notebooks in our teaching.\n\nCase study 1: Modern Information Engineering (MIE) \nThe Modern Information Engineering module is a new 15-credit master\u2019s level optional course unit that was proposed to model the process of modern software development using the Scrum framework[22] from the Agile software development methodology.[23] The unit was delivered in a blended format with both distance learning and a three-day block of face-to-face teaching sessions. Students (n = 21) were from a variety of backgrounds. Nine (43%) were NHS Graduate Management trainees. A further four (19%) had clinical backgrounds. The rest (38%) had a variety of backgrounds. The course ran over a nine-week period, with students working in Agile teams to add functionality to a medication prescribing dashboard (Figure 13) written in Python using the Flask web framework.[24] Students work in Sprints (two-week cycles) to add features to the dashboard, of which the skeleton code was provided to the groups.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 13 Example of the prescribing dashboard the teams would add functionally to following the Scrum framework.\n\n\n\nThe first part of this unit involved teaching fundamental Python and databases skills using SQL to students, many of whom have had no or limited exposure to computer programming (coding). We implemented the teaching of Python and SQL in the Jupyter Notebook environment. The unit was a module available as part of the master's module on a health informatics program that is a joint award between the University of Manchester and University College London (UCL).[25] The principal challenge faced was delivering teaching of coding skills to those who have little or no coding experience via distance learning in a way that allowed them to focus on obtaining these fundamental skills in the chosen language (Python) without introducing any additional complexity to the process. A further challenge was that unlike undergraduate courses where we may teach in person using a PC cluster (computer room\/lab) with preloaded software managed by IT services, many master\u2019s level students will be required (and usually prefer) to use their own computing devices (e.g., laptops, tablets, desktops). Supporting the use of software on these different operating systems and platforms adds an additional challenge. In order to remove or reduce these barriers to learning, we decided to make use of the interactive Jupyter Notebooks that support among others the Python programming language. We were then able to host a set of notebooks taking students through the various coding topics in order (Figure 14). A link to the notebooks was provided on the VLE for the module (i.e., Moodle) and the universities central username and password system added to prevent non-university affiliated personnel accessing the notebooks.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 14 List of notebooks covering the various topics of programming with Python.\n\n\n\nThe main reason we built our own notebooks rather than link to other existing resources (e.g., Software Carpentry) was to provide specific health-related examples for the students so that the domain would be familiar to them. Many of the computer science examples can be abstract in nature. By providing concrete health examples, it was hoped that this would help the students to see the relevance of potential applications of programming in health care settings. We were also able to add tasks throughout the notebooks that allowed students to code in the notebook and then view a model answer (e.g., Figure 15) using the exercise extension discussed previously.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 15 Example of task from notebook. Clicking the \u201cShow Solution\u201d button reveals the model answer.\n\n\n\nFigure 16 shows an example of a notebook from the set about the topic of variables and strings.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 16 Example of notebook on variables and strings programming topics.\n\n\n\nUsing the notebooks in this way allowed us to sidestep the issues of asking the students to download and set up Python on their machines with the associated complexities of supporting this. We do this later on in the unit where we move to group work and using an integrated development environment (IDE). At the beginning of the unit, we remove this barrier and allow the students to focus on learning the Python language and programming fundamentals. Initial feedback suggests that this improved their confidence with coding prior to the summative portion of the module. We provided support for students using the notebooks through the VLE and also by Slack (a cloud-based instant messaging service). Teaching assistants (TAs) would monitor the Slack channels and respond to issues the students faced with running the notebooks and Python in general.\n\nCase study 2: Introduction to Programming \nA second case study involving the teaching of basic coding in Python was a 15-credit module in a new distance learning PG Cert in clinical bioinformatics. It was designed to teach the fundamentals of genomics medicine to a diverse cohort of students. Clinical bioinformatics is a relatively new profession and represents the marriage of computer science with clinical practice. The computational and data skills needed to become a clinical bioinformatician are in short supply in the NHS, with training and education trying to fill the skills gap.[26] Those new to the field could come from many backgrounds, for example, those from the health sector with little or no programming experience to those with IT knowledge but with limited clinical experience.\nSimilarly, to case study 1, the module also adhered to agile principles but was delivered entirely online. The first part of the unit involved teaching basic GitHub and Python skills to students with differing levels of programming experience. It needed to support these varied learning requirements but also support students remotely, without face-to-face contact, while emulating clinical bioinformatics in practice. We therefore created an immersive and realistic software development environment with real-world practice-based problems in the form of sprints. To ensure an authentic learning experience, the students were taught to use Anaconda to install Python 3 onto their own machines. They also installed Git, and Windows users also installed and initialised Git Bash so that all students could be taught in a Linux environment. The course content was delivered to the students outside of Blackboard (learning management system) using GitHub.\nOther than initial introductory materials, the course material was taught using Jupyter Notebooks. The notebooks allowed us to provide interactive teaching on the basic principles of Python programming, including exercises that the students could complete within the notebook to hone their skills. Once the basic principles of Python programming were covered, we introduced the students to representational state transfer (REST) application programming interfaces (APIs) commonly used to collate genomics data. The immersive nature of the notebooks allowed us to build authentic tutorials to help students understand how data are retrieved from REST APIs and how they could build their own REST APIs. The notebooks gave the students the space to practice and develop these new skills comfortably in a fail-safe environment while using real-world examples. The flexibility of the notebooks also meant we could reuse them easily and incorporate slightly different examples to support the diverse student cohort.\nThe notebooks introduced the team-based sprint scenarios requiring the students to prototype code that will meet real-world needs of NHS scientists, as well as an in-production genomics software application, VariantValidator.[27] The interactive and engaging teaching provided by the notebooks helped scaffold the learning with short snippets of interactive code. These blocks of learning eventually culminated in a final SPRINT project where the learners built resources based on needs from their own practice (or became additional prototypes to support the VariantValidator project).\nOther tools such as Slack helped with the group work and educational support, such as solving initial configuration issues, pastoral support, and providing personal feedback on SPRINT activities. This peer-supported learning approach helped hone another essential skill in clinical bioinformatics, like being an active member of a community of practice.[28] It was the dual approach of active learning materials providing a fail-safe environment in the notebooks, coupled with the peer-supported learning via Slack; this meant we were able to deliver effective training into multiple countries, including a student working in frontline healthcare in China during the peak of the COVID-19 epidemic. At the end of the course, because the notebooks were downloaded to the students\u2019 machines, they had the tools, tutorials, and examples at their fingertips to learn back in practice. The aim of this \u201csandbox\u201d of editable and authentic learning materials was to help students to strengthen their programming skills in the long term and progress as members of the wider clinical bioinformatics community.\n\nEvaluation \nA detailed evaluation is beyond the scope of this paper as we are yet to run the various modules for significant time to collect sufficient data. We do, however, present some initial findings from a survey carried out on units using Jupyter Notebooks for teaching, as well as some statements from students about their experiences. Twelve students completed the survey and were asked six questions concerning the use of Jupyter Notebooks. Those six questions were:\n\n How useful did you find this course unit? (1 = not at all, 10 = very useful)\n How easy was it to use the Jupyter Notebooks in your learning? (1 = very difficult, 10 = very easy)\n Did the notebooks structure and combination of activities help you build understanding? (no\/yes)\n Did the pace of activities feel right to you? (no\/yes)\n How likely would you be to recommend Jupyter Notebooks and the learning approach we have followed? (1 = not at all likely, 10 = extremely likely)\n Overall, how satisfied were you with the course? (1 = least happy, 10 = happiest)\nThe results can be seen in Fig 17. We found that students provided predominantly positive responses to the questions asked. Results show that the students indicated that they would recommend notebooks for learning, found the course unit useful, and were satisfied with the course. For case study 2, students also provided reflective videos and feedback. This included the following comments on the practice-focused Jupyter Notebooks:\n\n\"[\u2026] the programming module starts with the basics for students where it is new to them. It gives an excellent overview of the different methodologies and languages and resources that are key to bioinformatics and what\u2019s also really helpful, or I found helpful, is that the code is taught in snippets in Juptyer Notebooks so you are able to try out small parts of the code for yourself \u2026 before you even need to get to grips with the development environment. So that was really useful.\"\n\r\n\u2014student on PG Cert clinical bioinformatics in their video feedback of the course (at 2 minutes 15 seconds)\n\u201cAs an NHS clinician with very little experience of coding, the course and specifically the introduction to programming has a steep learning curve. The modules have all been challenging but the accessibility of tutor support and their proactive approach to supporting students has meant that I\u2019ve never felt lost. As a non-specialist in this field, the course has provided me with the toolkit to understand the specific role that bioinformatics plays within the NHS. Whether one goes on undertake further study in this field or not, this PGCert course covers much of the material that a clinician will need familiarity with in the evolving healthcare landscape.\u201d\n\u201c[\u2026] very grateful for the quality of teaching on the course (across all the modules).\u201d\n\r\n\u2014student on PG Cert clinical bioinformatics, who created a reflective presentation on Introduction to Programming\n(Permission was obtained from students to use their statements and videos in publications and for marketing purposes. Videos are publically available on YouTube.)\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 17 Results of notebook student survey (n = 12).\n\n\n\nDiscussion \nOne has to be cautious when introducing technology like Jupyter Notebook into the classroom, especially when running a distance learning or online course if a large part of the course unit is dependent on notebook content. Such notebooks should be tested thoroughly and technical support available for their maintenance and any issues that may arise. Their use may also be more or less problematic with different user groups. Although it is more likely that those from a science, technology, engineering and math (STEM) background will be more comfortable with such tools, we cannot assume that this is necessarily the case. Myths like that of the \u201cdigital native\u201d (those born in the age of pervasive digital media) having some special advantages over other generations have been proven to be an unhelpful stereotype.[29] This means that one has to provide adequate support for the use of such tools to ensure their smooth adoption catering for different levels of digital literacy. Support for students with accessibility needs is also a consideration, and where possible, web content should conform to Web Content Accessibility Guidelines (WCAG).[30] To achieve these aims, it is useful to place the students at the center of the design process, considering who the target audience is and their needs, the application of the desired learning principles, how they will be presented, and importantly, how this design stands the test of time and is able to be adapted to meet changing needs.[31]\nFor the MIE module (case study 1), we were careful not to assume prior knowledge, especially given the diverse nature of the backgrounds and experience of health informatics students enrolled on that module. As this was applied to a blended module, we could not make use of standard computer clusters with preloaded software. Most of the students would be accessing the module using their own computing devices; therefore, we wanted to avoid the setup issues of downloading and installing a Python distribution (at least initially) until they had gained some familiarity and confidence with coding. We also didn\u2019t want to introduce an IDE at the initial stage of the module or use the console, as this is not ideal for writing larger blocks of code. These issues were overcome by remotely hosting the notebooks using cloud services and providing a link for the students to log in via the main university login system. This way, they would have their own secure copy of the notebooks for the module that could be accessed and modified, allowing them to rapidly focus on writing and learning to code, rather than all of the peripheral setup requirements and support issues.\nIn contrast, the approach for the Introduction to Programming module was to provide an authentic and immersive learning journey. The aim was to try to simulate everyday clinical bioinformatics working practices but in a safe learning environment that could give them the space to fail and learn from their mistakes both individually and as a team. This meant students needed to download the notebooks locally, work with different versions using GitHub, and work as a team on Slack. The challenge was to provide enough support to help deal with any issues they had with the tools and techniques being taught, but with enough autonomy so they could develop much-needed problem-solving skills in clinical bioinformatics. This balancing act required a lot of resource both at the design stage, with additional materials for different learner requirements, and during the delivery stage.\nSupporting modules such as these may require more resources and support than traditional face-to-face modalities. The interactive coding tasks helped the student to gain hands-on experience in coding while customising their own class notes, which they could download and keep beyond the duration of the course. Such tasks and interactive elements provided via notebooks uniquely helped to move students from a more static learning experience into a more dynamic experience.[17] This can provide a deeper level of immersion in the tasks (for example, exploring a dataset via an interactive plot or applying skills to a practice-based problem of their choosing). In terms of Bloom's (revised) taxonomy, this moves students towards the top of higher ordered thinking into creation and production.[32]\nCareful consideration should also be afforded to the overall aims of the module, unit, or course, and technology should be used where appropriate to improve or facilitate learning, rather than be used for novelty purposes. Findings suggest that learning should be the main focus, rather than an aim to be \u201ctech-centric.\u201d[33] If we want to apply digital pedagogy successfully, we need to match each bit of technology with our required pedagogical goals.[33] Essentially, the technology is there to enhance the learning and should be chosen to support the fulfilment of the learning objectives while considering the students\u2019 wider contexts and learning environments. A further consideration is the quality and practices that we impart through such methods. There are calls for journal editors and reviewers to enforce computational reproducibility[34]; however, while many scientists use and write code on a regular basis, they often lack formal training in good software engineering practices.[35] To embed good practices that students can use in their further research careers, we need to ensure that whenever possible, the content we generate helps to distill these practices. This in turn negates the importance of providing appropriate training for educators themselves.\n\nConclusion \nThe use of Jupyter Notebooks on several of our university modules has been positively received by both staff and students, who see them as a useful resource for learning to code and communicate research findings and analysis, and in the cases presented, learning the Python programming language specifically. The use of notebooks in such units also gives students an introduction to the notebook environment, which some may go on to use for research purposes later in their career or for the research component of their masters degrees. The use of digital notebooks and other technologies should be carefully evaluated to ensure they add real value to the learning aims and objectives, placing the pedagogic aims of the course at the centre of the process. Given that the use of such tools is becoming more ubiquitous in the bioscience research and scientific education domains, it would be advantageous for academic tutors in such fields to have an awareness and understanding of their application and to consider their use for providing interactive components to computational learning tasks where appropriate.\n\nAcknowledgements \nFunding \nThere was no funding for this project.\n\nCompeting interests \nThe authors have declared that no competing interests exist.\n\nReferences \n\n\n\u2191 Gregory, J.; Salmon, G. (2013). \"Professional development for online university teaching\". Distance Education 34 (3): 256\u201370. doi:10.1080\/01587919.2013.835771.   \n\n\u2191 Georgina, D.A.; Olson, M.R. (2008). \"Integration of technology in higher education: A review of faculty self-perceptions\". The Internet and Higher Education 11 (1): 1\u20138. doi:10.1016\/j.iheduc.2007.11.002.   \n\n\u2191 3.0 3.1 \"Jupyter\". Project Jupyter. https:\/\/jupyter.org\/ . Retrieved 27 January 2020 .   \n\n\u2191 4.0 4.1 Richardson, M.L.; Amini, B.. \"Scientific Notebook Software: Applications for Academic Radiology\". Current Problems in Diagnostic Radiology 47 (6): 368\u201377. doi:10.1067\/j.cpradiol.2017.09.005. PMID 29122394.   \n\n\u2191 Cone, M.. \"Markdown Guide\". https:\/\/www.markdownguide.org\/ . Retrieved 23 September 2019 .   \n\n\u2191 \"An introduction to LaTeX\". The LaTeX Project. https:\/\/www.latex-project.org\/about\/ . Retrieved 27 January 2020 .   \n\n\u2191 Roberts, A.; Oetiker, T.; Partl, H. et al.. \"LaTeX\/Mathematics\". LaTeX - WikiBooks. https:\/\/en.wikibooks.org\/wiki\/LaTeX\/Mathematics . Retrieved 27 January 2020 .   \n\n\u2191 \"Anaconda Distribution\". Anaconda, Inc. 2019. Archived from the original on 06 September 2019. https:\/\/web.archive.org\/web\/20190906215334\/https:\/\/www.anaconda.com\/distribution\/ . Retrieved 21 September 2019 .   \n\n\u2191 \"JupyterHub\". Project Jupyter. https:\/\/jupyter.org\/hub . Retrieved 27 January 2020 .   \n\n\u2191 \"Using Interact\". ipywidgets User Guide. Project Jupyter. https:\/\/ipywidgets.readthedocs.io\/en\/latest\/examples\/Using%20Interact.html .   \n\n\u2191 \"What researchers think about the culture they work in\". Wellcome Trust. 14 January 2020. https:\/\/wellcome.org\/reports\/what-researchers-think-about-research-culture .   \n\n\u2191 Hardwicke, T.E.; Mathur, M.B.; MacDonald, K. et al. (2018). \"Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition\". Royal Society Open Science 5 (8): 180448. doi:10.1098\/rsos.180448.   \n\n\u2191 \"U.K. Reproducibility Network\". U.K. Reproducibility Network. https:\/\/www.ukrn.org\/ .   \n\n\u2191 Munaf\u00f2, M. (2019). \"Raising research quality will require collective action\". Nature 576: 183. doi:10.1038\/d41586-019-03750-7.   \n\n\u2191 Rule, A.; Birmingham, A.; Zuniga, C. et al. (2018). \"Ten Simple Rules for Reproducible Research in Jupyter Notebooks\". arXiv: 1\u20138. https:\/\/arxiv.org\/abs\/1810.08055 .   \n\n\u2191 Medley, J.K.; Choi, K.; K\u00f6nig, M. et al. (2018). \"Tellurium notebooks\u2014An environment for reproducible dynamical modeling in systems biology\". PLOS Computational Biology 14 (6): e1006220. doi:10.1371\/journal.pcbi.1006220.   \n\n\u2191 17.0 17.1 Barba, L.A.; Barker, L.J.; Blank, D.S. et al. (06 December 2019). \"Teaching and Learning with Jupyter\". GitHub. https:\/\/jupyter4edu.github.io\/jupyter-edu-book\/index.html . Retrieved 27 January 2020 .   \n\n\u2191 \"nbgrader\". Jupyter Development Team. https:\/\/nbgrader.readthedocs.io\/en\/stable\/ . Retrieved 27 January 2020 .   \n\n\u2191 \"binder\". Project Jupyter. https:\/\/mybinder.org\/ . Retrieved 05 June 2020 .   \n\n\u2191 20.0 20.1 Bussonnier, M.; Forde, J.; Freeman, J. et al. (2018). \"Binder 2.0 - Reproducible, interactive, sharable environments for science at scale\". Proceedings of the 17th Python in Science Conference: 113\u201320. doi:10.25080\/Majora-4af1f417-011.   \n\n\u2191 Data Carpentry. \"Sharing Jupyter Notebooks\". GitHub. https:\/\/reproducible-science-curriculum.github.io\/sharing-RR-Jupyter\/ . Retrieved 05 June 2020 .   \n\n\u2191 \"What Is Scrum?\". Advanced Development Methods, Inc. https:\/\/www.scrum.org\/resources\/what-is-scrum . Retrieved 25 January 2020 .   \n\n\u2191 \"Agile 101\". Agile Alliance. https:\/\/www.agilealliance.org\/agile101\/ . Retrieved 25 January 2020 .   \n\n\u2191 \"Flask Documentation\". Pallets. https:\/\/flask.palletsprojects.com\/en\/1.1.x\/ . Retrieved 11 March 2020 .   \n\n\u2191 \"MSc\/PGDip\/PGCert Health Informatics (UCL\/UoM Joint Award) \/ Overview\". University of Manchester. https:\/\/www.manchester.ac.uk\/study\/masters\/courses\/list\/12478\/msc-pgdip-pgcert-health-informatics-ucl-uom-joint-award\/ . Retrieved 27 January 2020 .   \n\n\u2191 Attwood, T.K.; Blackford, S.; Brazas, M.D. et al. (2019). \"A global perspective on evolving bioinformatics and data science training needs\". Briefings in Bioinformatics 20 (2): 398\u2013404. doi:10.1093\/bib\/bbx100. PMC PMC6433731. PMID 28968751. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6433731 .   \n\n\u2191 Freeman, P.J.; Hart, R.K.; Gretton, L.J. et al. (2018). \"VariantValidator: Accurate validation, mapping, and formatting of sequence variation descriptions\". Human Mutation 39 (1): 61\u201368. doi:10.1002\/humu.23348.   \n\n\u2191 Davies, A.C.; Harris, D.; Banks-Gatenby, A. et al. (2019). \"Problem-based learning in clinical bioinformatics education: Does it help to create communities of practice?\". PLOS Computational Biology 15 (6): e1006746. doi:10.1371\/journal.pcbi.1006746. PMC PMC6597031. PMID 31246944. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6597031 .   \n\n\u2191 Kirschner, P.A.; De Bruyckere, P. (2017). \"The myths of the digital native and the multitasker\". Teaching and Teacher Education 67: 135\u201342. doi:10.1016\/j.tate.2017.06.001.   \n\n\u2191 Kirkpatrick, A.; O'Connor, J.O.; Campbell, A. et al. (05 June 2018). \"Web Content Accessibility Guidelines (WCAG) 2.1\". W3C. https:\/\/www.w3.org\/TR\/WCAG21\/ . Retrieved 25 January 2020 .   \n\n\u2191 Beetham, H.; Sharpe, R., ed. (2013). Rethinking Pedagogy for a Digital Age (2nd ed.). Routledge. ISBN 9780415539975.   \n\n\u2191 Anderson, L.W.; Krathwohl, D.R., ed. (2001). Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives (1st ed.). Pearson. ISBN 978-0801319037.   \n\n\u2191 33.0 33.1 Curry, N. (05 October 2018). \"Putting the pedagogy first in digital pedagogies\". World of Better Learning Blog. https:\/\/www.cambridge.org\/elt\/blog\/2018\/10\/05\/putting-the-pedagogy-first-in-digital-pedagogies\/ . Retrieved 27 January 2020 .   \n\n\u2191 Gymrek, M.; Farjoun, Y. (2016). \"Recommendations for open data science\". GigaScience 5: 22. doi:10.1186\/s13742-016-0127-4. PMC PMC4870738. PMID 27195107. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4870738 .   \n\n\u2191 Wilson, G.; Aruliah, D.A.; Brown, C.T. et al. (2014). \"Best practices for scientific computing\". PLOS Biology 12 (1): 1001745. doi:10.1371\/journal.pbio.1001745. PMC PMC3886731. PMID 24415924. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3886731 .   \n\n\nNotes \nThis presentation attempts to remain faithful to the original, with only a few minor changes to presentation. Grammar and punctuation has been updated reasonably to improve readability. In some cases important information was missing from the references, and that information was added. The original URL for the Anaconda Distribution is dead; an archived version of the URL was used for this version. The original \"Using Interact\" URL was also broken; an updated live version was substituted for this version. The UKRN URL also changed, and a current URL is used for this version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\">https:\/\/www.limswiki.org\/index.php\/Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on educationLIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 22 May 2021, at 23:41.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 592 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","31bea4f4ab2f82a24f0d0dad004bfbb0_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Using_interactive_digital_notebooks_for_bioscience_and_informatics_education skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Using interactive digital notebooks for bioscience and informatics education<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Interactive digital notebooks provide an opportunity for researchers and educators to carry out data analysis and report results in a single digital format. Further to just being digital, the format allows for rich content to be created in order to interact with the code and data contained in such a notebook to form an educational narrative. This primer introduces some of the fundamental aspects involved in using <a href=\"https:\/\/www.limswiki.org\/index.php\/Jupyter_Notebook\" title=\"Jupyter Notebook\" class=\"wiki-link\" data-key=\"26fd35430c10e009a142bbab5dbf617a\">Jupyter Notebook<\/a> in an educational setting for teaching in the <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">health informatics<\/a> disciplines. We also provide two case studies that detail 1. how we used Jupyter Notebooks to teach non-coders programming skills on a blended master\u2019s degree module for a health informatics program, and 2. a fully online distance learning unit on programming for a postgraduate certificate (PG Cert) in clinical bioinformatics, with a more technical audience.\n<\/p><p><b>Keywords<\/b>: bioinformatics, health informatics, programming, data analysis, Jupyter Notebook, education\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Universities and other higher education institutions are now under increasing pressure to provide more online and distance learning courses and to deliver them cost effectively and rapidly.<sup id=\"rdp-ebb-cite_ref-GregoryProf13_1-0\" class=\"reference\"><a href=\"#cite_note-GregoryProf13-1\">[1]<\/a><\/sup> This increase in demand is partly based on students wanting more flexible study options in comparison to traditional higher education course delivery to aid in study around employment and family commitments. This is also driven by financial considerations that allow higher education institutions to scale course delivery while managing infrastructural provision (e.g., access to rooms for teaching and limited capacity for face-to-face delivery).<sup id=\"rdp-ebb-cite_ref-GeorginaInteg08_2-0\" class=\"reference\"><a href=\"#cite_note-GeorginaInteg08-2\">[2]<\/a><\/sup> To meet this challenge, we require tools that cater for students with varying levels of digital literacy and reduce the burden of them having to download and install software, all of which requires support, which is more difficult to provide at a distance. This can be further complicated when students use managed equipment (e.g., National Health Service [NHS] employees) and may not have administrator rights to install software.\n<\/p><p>Digital notebooks provided us with a way of meeting these needs, as they are easy to set up, straightforward to use, and can support rich and interactive content. Here, we present a primer on how to use digital notebooks (specifically <a href=\"https:\/\/www.limswiki.org\/index.php\/Jupyter_Notebook\" title=\"Jupyter Notebook\" class=\"wiki-link\" data-key=\"26fd35430c10e009a142bbab5dbf617a\">Jupyter Notebooks<\/a>) for teaching and assessment, along with details of two case studies where we used notebooks to teach Python programming and database skills for clinical <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">health informatics<\/a> students of varying levels of technical experience. The case studies and methods presented can be applied to both distance learning and face-to-face teaching scenarios.\n<\/p><p>We will start by covering what a Jupyter Notebook is along with the different \u201ccell\u201d types available. We then look at how they can be run and enhanced with extensions to add items like exercise tasks and other interactivity before looking at how they can be used in assessment. Next, we present two case studies where we have applied notebooks to teach different groups of students to give some examples of the different contexts they can be used in. Finally, we end with a discussion to synthesise our experiences of using notebooks to educate students and their further potential, with considerations for education.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"What_is_a_Jupyter_Notebook.3F\">What is a Jupyter Notebook?<\/span><\/h2>\n<p>Jupyter Notebook is an open-source web application that runs in an internet browser. It allows the sharing of code, <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">data analysis<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_visualization\" title=\"Data visualization\" class=\"wiki-link\" data-key=\"4a3b86cba74bc7bb7471aa3fc2fcccc3\">data visualizations<\/a> (which can be interactive), math formulas, and other embedded media (e.g., YouTube videos, images, and web links), all in a single document combining interactive and narrative components. This takes the form of a document that is composed of multiple cells that encapsulate the content of the notebook (Figure 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"148b5f2e010e52caca2b4e43f3c3c54b\"><img alt=\"Fig1 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/1a\/Fig1_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> A new Python 3 notebook with three empty cells denoted by the grey rectangles. The currently selected cell is highlighted in green.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Jupyter notebooks were created by Project Jupyter, which, according to their website, states that \u201cProject Jupyter exists to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages.\u201d<sup id=\"rdp-ebb-cite_ref-JupyterHome_3-0\" class=\"reference\"><a href=\"#cite_note-JupyterHome-3\">[3]<\/a><\/sup> This includes various standards for interactive computing, including the notebook document format that is based on JavaScript Object Notation (JSON). The name Jupyter is composed of the initial three languages supported: Julia, Python, and <a href=\"https:\/\/www.limswiki.org\/index.php\/R_(programming_language)\" title=\"R (programming language)\" class=\"wiki-link\" data-key=\"1b0aa598f071aca4c5b4ee08d8bb2bde\">R<\/a>.<sup id=\"rdp-ebb-cite_ref-RichardsonScien18_4-0\" class=\"reference\"><a href=\"#cite_note-RichardsonScien18-4\">[4]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Anatomy_of_a_notebook\">Anatomy of a notebook<\/span><\/h2>\n<p>Jupyter notebooks are available in various programming languages, with current support for over 40 different programming languages.<sup id=\"rdp-ebb-cite_ref-JupyterHome_3-1\" class=\"reference\"><a href=\"#cite_note-JupyterHome-3\">[3]<\/a><\/sup> These include the popular languages used for data science, such as Julia, Python, and R (Figure 2).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"c8395f33d47389a9008b5568f51c4937\"><img alt=\"Fig2 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/84\/Fig2_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> A simple function that returns the value of the sum of two numbers showing different kernels (programming languages) in the notebooks; this example shows Python (left), Julia (middle), and R (right).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The notebooks are made up of units called \u201ccells\u201d that can be executed (run) in order to render their contents in different ways.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Cell_types\">Cell types<\/span><\/h3>\n<p>There are two principle cell types. The first cell type is the \u201cMarkdown\u201d cell, which is used to present text, images, equations, and other resources. The second cell type is the \u201ccode\u201d cell, which allows the user to enter code written in a chosen programming language that will execute in the notebook. To execute the contents of any cell, the user can press the Shift and Enter keys together, or alternatively click on the \u201cRun\u201d button in the main menu bar across the top of the screen. If the cell being run is a code cell, it will cause the code in the cell to be executed and any output displayed immediately below it. This is indicated by the \u201cIn\u201d and \u201cOut\u201d words located to the left of the cells, as seen in Fig 2.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Styling_cells\">Styling cells<\/span><\/h3>\n<p>Markdown cells can be styled with Markdown, which is a lightweight mark-up language for styling text.<sup id=\"rdp-ebb-cite_ref-MGMarkdown_5-0\" class=\"reference\"><a href=\"#cite_note-MGMarkdown-5\">[5]<\/a><\/sup> This works by turning Markdown text into HTML (Figure 3).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"647ed711e14c6f7c6e5dbbfca7ef22ba\"><img alt=\"Fig3 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/89\/Fig3_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3<\/b> Example of a markdown cell (left) and the output of the styled cell when the cell is run (right).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>These cells can also display plain text as output with no styling. Another useful feature for teaching math-based courses or sharing formulas, etc. is the integration of LaTeX support. LaTeX is a popular typesetting document preparation system [6] that was built on the Tex typesetting language, originally developed by the American computer scientist Donald Knuth.<sup id=\"rdp-ebb-cite_ref-LatexAbout_6-0\" class=\"reference\"><a href=\"#cite_note-LatexAbout-6\">[6]<\/a><\/sup> LaTeX is widely used by the scientific community (e.g., computer scientists) to write academic publications (journal and conference papers). LaTeX math notation can be added to markdown cells to display formulas using common math notation. For example, the code below produces the output seen in Figure 4:\n<\/p><p><tt>$\u2009$\n<\/p><p>\\sigma = \\sqrt{\\frac{1}{N}\\sum_{i = 1}^{N} (x_i-\\mu)^2}\n<\/p><p>$\u2009$<\/tt>\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"34e178db9fa3b3ffc7fae575f0db4c11\"><img alt=\"Fig4 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/90\/Fig4_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 4<\/b> Output of LaTeX math notation producing the formula for the population standard deviation.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The LaTeX wikibook math section<sup id=\"rdp-ebb-cite_ref-WBLaTeX_7-0\" class=\"reference\"><a href=\"#cite_note-WBLaTeX-7\">[7]<\/a><\/sup> is a useful resource for learning about the math notation options available in LaTeX. Table 1 provides an overview of some of the useful Python libraries for numerical and scientific computing that can be incorporated into the notebook environment.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"b4025ff3836db0038c1aa9fd054d46c3\"><img alt=\"Tab1 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/09\/Tab1_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 1.<\/b> Some useful Python libraries for numerical and scientific computing.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Running_Jupyter_Notebooks\">Running Jupyter Notebooks<\/span><\/h2>\n<p>There are different ways of accessing Jupyter Notebooks. The Anaconda Distribution<sup id=\"rdp-ebb-cite_ref-AnacondaDist_8-0\" class=\"reference\"><a href=\"#cite_note-AnacondaDist-8\">[8]<\/a><\/sup>, a data science platform for Python and R, provides a free Python distribution, which includes Jupyter Notebooks. Another option includes JupyterHub<sup id=\"rdp-ebb-cite_ref-JupyterHub_9-0\" class=\"reference\"><a href=\"#cite_note-JupyterHub-9\">[9]<\/a><\/sup>, which is designed for groups of users to access notebooks on the <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud<\/a> or locally hosted and maintained on their own devices. Once run, the user is greeted with a page showing the various files and folders available (Figure 5). Selecting the \u201cnew\u201d option from the menu allows the user to create a new notebook in the selected language; alternatively, an existing notebook (ipynb) file can be loaded by selecting the required file from the list of files in the main list to the left of the screen.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"a43412963a3cb4bdbdc406041e608b59\"><img alt=\"Fig5 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ea\/Fig5_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 5<\/b> The files and folders tab seen when launching Jupyter notebooks locally. A new notebook is created by selecting the new dropdown option and choosing the required language.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Jupyter_Notebooks.2C_JupyterLab.2C_and_JupyterHub\">Jupyter Notebooks, JupyterLab, and JupyterHub<\/span><\/h2>\n<p>Project Jupyter has created several resources and services surrounding the initial notebooks. This can sometimes cause some confusion among beginners. The difference among them are briefly described here.\n<\/p>\n<ol><li> Jupyter Notebook is an interactive computational web application that combines code, text, data analysis, and other media in a single document.<\/li>\n<li> JupyterLab builds on the original Jupyter Notebook to provide an online interactive development environment that allows users to access notebooks with data and file viewers, text editors, and terminals all in the same environment. This helps to better integrate notebooks with other documents and resources in a single environment.<\/li>\n<li> JupyterHub allows multiple users (groups) to access notebooks and other resources. This can be useful for students and companies that want one or more groups to access and use a computational environment and resources without having to install and set things up. The management of these groups can be carried out by system administrators. Individual notebooks and the JupyterLab can be accessed via the Hub. The Hub can be run in the cloud or on a group's own hardware.<\/li><\/ol>\n<p>As these offerings build on the initial notebook and have notebooks at their core, this article describes the notebooks for beginners, rather than the additional platforms and services that incorporate them. Notebooks themselves work in a similar way regardless of being accessed alone or via JupyterLab or JupyterHub. It is worth being aware of these options, however, for building and sharing resources around the notebooks that you may develop.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Notebook_extensions\">Notebook extensions<\/span><\/h2>\n<p>A number of different \u201cbolt on\u201d extensions exists for the notebooks. These can be extremely useful for including additional features into a notebook. Some examples include the ability to split a cell into two different cells horizontally, a spellchecker, auto-numbering of equations, and an extension for making exercise tasks (discussed later). To enable and utilize the additional features that are available with the notebooks, the following commands should be entered into the command prompt (e.g., the Anaconda prompt or Powershell):\n<\/p><p><tt>pip install jupyter_contrib_nbextensions\n<\/p><p>jupyter contrib nbextension install\u2014user\n<\/p><p>pip install jupyter_nbextensions_configurator\n<\/p><p>jupyter nbextensions_configurator enable\u2013user<\/tt>\n<\/p><p>This enables the \u201cNBextensions\u201d tab (Figure 6). When clicked on, the user is presented with a series of checkboxes for the various extensions. There is also a description, often with associated screenshots and\/or animations previewing what the extension does.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"918ecd44d96af4586c78843ab96351ce\"><img alt=\"Fig6 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/70\/Fig6_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 6<\/b> The NBextensions tab for selecting the various notebook extensions.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>When a new notebook is opened, the selected extensions appear as small icon buttons under the main menu (Figure 7).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"dd710d1a718a8aa0088aabc801b1f55b\"><img alt=\"Fig7 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/da\/Fig7_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 7<\/b> Enabled notebook extension icons shown in red box.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Magic_commands\">Magic commands<\/span><\/h2>\n<p>IPython (the \u201cInteractive Python\u201d kernel used in Jupyter Notebook) also supports what are known as magic commands or functions, which are used to change the standard behaviour of IPython. Magic commands come in two different types: \u201cline\u201d and \u201ccell\u201d magics. The <tt>%lsmagic<\/tt> command displays a list of all the available line magics, while <tt>%magic<\/tt> displays a help window with information about magic functions. A line magic only works on the line of code that it precedes, whereas a cell magic applies the function to the entire cell. A line magic is prefixed with a single percentage character (%), whereas a cell magic is prefixed with two percent characters (%%). Figure 8 shows an example of this, where we use the magic functions to load a Structured Query Language (SQL) extension and specify a database engine such as SQLite. The second code cell employs cell magic to allow us to write and execute SQL commands in the notebook environment to create a database table. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"7617e902eae59b40c4ff4f5ca5b99278\"><img alt=\"Fig8 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ee\/Fig8_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 8<\/b> Line and cell magic\u2019s used to add SQL (Structured Query Language) functionality to a Python notebook.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Widgets\">Widgets<\/span><\/h2>\n<p>Widgets can be used for interactive elements in notebooks.<sup id=\"rdp-ebb-cite_ref-PJUsing_10-0\" class=\"reference\"><a href=\"#cite_note-PJUsing-10\">[10]<\/a><\/sup> Figure 9 shows an example of this where the \u201cinteract\u201d function runs the \u201cget_val\u201d function displaying a slider with the default value (5 in this case) selected. The user can then change the value by moving the slider to the left or right. Figure 10 shows another example, this time using a drop-down list of options created from a Python list.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig9_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"6efbf65e63ba9912af6da4013a8bd46c\"><img alt=\"Fig9 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/9c\/Fig9_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 9<\/b> Example of notebook interaction.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig10_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"041b7a458ba21794ab6e653020780eaf\"><img alt=\"Fig10 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/64\/Fig10_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 10<\/b> An interactive drop-down list created using a Python list.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>A more substantive example of using interactive widgets is highlighted by Richardson and Behrang, who use Python notebooks to view Digital Imaging and Communications in Medicine (<a href=\"https:\/\/www.limswiki.org\/index.php\/DICOM\" title=\"DICOM\" class=\"wiki-link\" data-key=\"f0c7c747895286ff8785b6ed4dbc7ec0\">DICOM<\/a>) images.<sup id=\"rdp-ebb-cite_ref-RichardsonScien18_4-1\" class=\"reference\"><a href=\"#cite_note-RichardsonScien18-4\">[4]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"How_Jupyter_enhances_collaboration_and_reproducibility\">How Jupyter enhances collaboration and reproducibility<\/span><\/h2>\n<p>Reproducibility in science is an important concept. Without reproducability, there is a lack of transparency about what was done. One would expect that if scientists follow the same method, the results will be the same. This is sometimes difficult to achieve with complex data and analysis methods. The quality of research in relation to collaboration was brought into question in a recent Wellcome Trust report on research culture that raised concerns over the impact of lack of research collaboration on research quality, and in some cases, unhealthy competition between researchers.<sup id=\"rdp-ebb-cite_ref-WTWhatRes20_11-0\" class=\"reference\"><a href=\"#cite_note-WTWhatRes20-11\">[11]<\/a><\/sup> As Hardwicke and colleagues highlight, the availability of data is essential for a self-correcting ecosystem in science, which can be undermined by unclear analysis and poorly curated data, which, in turn, impedes analytic reproducibility.<sup id=\"rdp-ebb-cite_ref-HardwickeData18_12-0\" class=\"reference\"><a href=\"#cite_note-HardwickeData18-12\">[12]<\/a><\/sup>\n<\/p><p>There has been a counter-movement to improve these issues by organizations such as the U.K. Reproducibility Network (UKRN)<sup id=\"rdp-ebb-cite_ref-UKRNHome_13-0\" class=\"reference\"><a href=\"#cite_note-UKRNHome-13\">[13]<\/a><\/sup>, which is a network of 10 universities in the U.K. that are concerned with reproducibility in research. The founder of UKRN calls for institutional changes to promote open-research practices.<sup id=\"rdp-ebb-cite_ref-Munaf.C3.B2Raising19_14-0\" class=\"reference\"><a href=\"#cite_note-Munaf.C3.B2Raising19-14\">[14]<\/a><\/sup> Although various research studies do share their data, other researchers\u2019 understanding of the shared dataset and their ability to repeat the previous analysis hinges on the documentation of both the dataset and analysis steps followed, as well as being able to replicate the software environment in order to run the code in the first instance. Because of these requirements, notebooks are being used increasingly by researchers to share analysis code along with an explanation and steps involved in processing the data for reproducible research purposes. This has led to widescale use in the research community.<sup id=\"rdp-ebb-cite_ref-RuleTen18_15-0\" class=\"reference\"><a href=\"#cite_note-RuleTen18-15\">[15]<\/a><\/sup> By using interactive notebooks, the data analysis code and steps taken can be shared together with any additional documentation, formulas, etc. that are required to understand the applied method. Sharing data and analysis code in such a way dramatically improves the speed in which the analysis can be rerun by other researchers. Researches are also building on notebook technology for novel purposes, for example, Tellurium notebooks that were developed to support the creation of reproducible models for systems and synthetic biology.<sup id=\"rdp-ebb-cite_ref-MedleyTellur18_16-0\" class=\"reference\"><a href=\"#cite_note-MedleyTellur18-16\">[16]<\/a><\/sup>\n<\/p><p>Aside from the research applications, Jupyter Notebooks are also being increasingly used to teach subjects like data science and programming<sup id=\"rdp-ebb-cite_ref-BarbaTeach19_17-0\" class=\"reference\"><a href=\"#cite_note-BarbaTeach19-17\">[17]<\/a><\/sup>, as they feature dynamic responses such as interactive visualizations and rapid updating of results based on the filtering of data (e.g., Figure 11).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig11_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"d7e520c5e8b4c1ca0f1ab97f1e6a3c91\"><img alt=\"Fig11 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/c1\/Fig11_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 11<\/b> Interactive plot generated with the \u201cplotly\u201d module that can be rotated and zoomed with individual data points selected.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Notebooks_and_assessment\">Notebooks and assessment<\/span><\/h2>\n<p>Notebooks can also be set up to carry out formative or summative assessment. The \u201cnbgrader\u201d tool<sup id=\"rdp-ebb-cite_ref-nbgrader_18-0\" class=\"reference\"><a href=\"#cite_note-nbgrader-18\">[18]<\/a><\/sup> allows for the creation and grading of assignments in the notebook environment. The tool allows a user to generate an instructor version of a notebook that has predefined solutions. This, in turn, is used to generate the student version of the notebooks without the solutions. These student versions of the notebook(s) can then be distributed to the students by email or via a virtual learning environment (VLE). The principal aims of the tool were to address issues surrounding the maintenance of separate student and instructor notebook versions, automatic grading of exercises, the manual grading of \u201cfree response\u201d questions, and the ability to provide feedback to students. There are two ways of using the nbgrader. The first is a standalone version, while the second is designed to work with JupyterHub, which can manage the release and collection of submitted assessments. The nbgrader adds a tool bar to each cell to make the cell either an \u201canswer\u201d or a \u201ctest\u201d cell. The answer cells allow students to add code placed between a placeholder. Unit tests are written by the instructor to evaluate the correctness of a student\u2019s solution. Tests can also be hidden from the students. Points can be assigned to each cell to assign specified marks if the unit tests pass. Cells can also be set to \u201cmanually graded\u201d answers so students can write free text, code formulas, etc. Student feedback can be provided when grading by adding text to any required cell and then converting the notebooks into HTML format so they can be emailed or added to VLE for the students to view.\n<\/p><p>A simpler method of providing interactive tasks for formative assessment that does not require the knowledge of writing unit tests is to use the exercise extension. This extension can be used to add exercises (Figure 12). Adding feedback in the form of exercises is a unique feature of notebooks that elevates them from simply being an online textbook. The ability to provide interactive tasks that let students engage directly with the notebook without the need to use additional software is a powerful feature. Moreover, this helps maintain the narrative flow, as the exercises can be woven into the content in appropriate places without diverting the user to other tools or resources, all of which helps with the overall user experience.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig12_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"ccbf3138076c3cfe334468a5aca5b70a\"><img alt=\"Fig12 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/c4\/Fig12_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 12<\/b> Example using the \u201cexercise2\u201d extension to create a task. When the \u201cshow solution\u201d button is pressed, the answer is displayed below.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Here, we give an example of an exercise where we create a task cell and put the solution in the preceding cell. The solution cell can be hidden until the \u201cshow solution\u201d button (Figure 12) is activated, which reveals the hidden cell. This is a good way of adding coding tasks for students and then presenting them with a model answer or solution for comparison and\/or further explanation. Figure 12 shows a task where the student has to add a textual value to a Python dictionary data structure and output the result. Students can attempt to write the code for this and then toggle the solution to check their answer with the one provided.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Sharing_your_notebooks\">Sharing your notebooks<\/span><\/h2>\n<p>Notebooks can be shared in the same way as any other file. In order to run a notebook, however, users will need to install and set up software (i.e., Anaconda). This may not be the ideal solution given that novices may have difficulties installing and setting up the environment required to view and use notebooks. This is further compounded if the user needs to install extra libraries and extensions that may be required to run a notebook as intended. One helpful way around this when sharing notebooks with novices is the Binder project.<sup id=\"rdp-ebb-cite_ref-Binder_19-0\" class=\"reference\"><a href=\"#cite_note-Binder-19\">[19]<\/a><\/sup>\n<\/p><p>Binder is a web service (currently open-source) that allows users to create interactive sharable and reproducible computational environments in the cloud.<sup id=\"rdp-ebb-cite_ref-BussonnierBinder18_20-0\" class=\"reference\"><a href=\"#cite_note-BussonnierBinder18-20\">[20]<\/a><\/sup> Binder uses several different technologies (i.e., repo2docker, JupyterHub, and BinderHub) that allow a user to place their notebooks in a repository (e.g., GitHub). Once done, a form can be filled in on the Binder website (mybinder.org). This includes a repository Uniform Resource Locator (URL), Git tag, and optional path of the notebook file. Following this, a user will receive a URL that they can send to others to share their notebooks.\n<\/p><p>For a more technical explanation of how Binder works, please see the Binder paper presented at the SCIPY conference in 2018 and its associated YouTube video.<sup id=\"rdp-ebb-cite_ref-BussonnierBinder18_20-1\" class=\"reference\"><a href=\"#cite_note-BussonnierBinder18-20\">[20]<\/a><\/sup> For more information on how to implement sharing Notebooks with Binder, see the Data Carpentry tutorial, which guides users through sharing their notebooks with GitHub and using Binder.<sup id=\"rdp-ebb-cite_ref-DCSharing_21-0\" class=\"reference\"><a href=\"#cite_note-DCSharing-21\">[21]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Case_studies\">Case studies<\/span><\/h2>\n<p>We present two short case studies detailing how we have used Jupyter Notebooks to teach programming skills to different audiences on two of our courses, a master's module on a health informatics program and an introduction to programming module on postgraduate certificate (PG Cert) in clinical bioinformatics. This is followed by a brief initial evaluation of the use of notebooks in our teaching.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Case_study_1:_Modern_Information_Engineering_.28MIE.29\">Case study 1: Modern Information Engineering (MIE)<\/span><\/h3>\n<p>The Modern Information Engineering module is a new 15-credit master\u2019s level optional course unit that was proposed to model the process of modern software development using the Scrum framework<sup id=\"rdp-ebb-cite_ref-ScrumWhatIs_22-0\" class=\"reference\"><a href=\"#cite_note-ScrumWhatIs-22\">[22]<\/a><\/sup> from the Agile software development methodology.<sup id=\"rdp-ebb-cite_ref-Agile101_23-0\" class=\"reference\"><a href=\"#cite_note-Agile101-23\">[23]<\/a><\/sup> The unit was delivered in a blended format with both distance learning and a three-day block of face-to-face teaching sessions. Students (<i>n<\/i> = 21) were from a variety of backgrounds. Nine (43%) were NHS Graduate Management trainees. A further four (19%) had clinical backgrounds. The rest (38%) had a variety of backgrounds. The course ran over a nine-week period, with students working in Agile teams to add functionality to a medication prescribing dashboard (Figure 13) written in Python using the Flask web framework.<sup id=\"rdp-ebb-cite_ref-FlaskDoc_24-0\" class=\"reference\"><a href=\"#cite_note-FlaskDoc-24\">[24]<\/a><\/sup> Students work in Sprints (two-week cycles) to add features to the dashboard, of which the skeleton code was provided to the groups.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig13_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"50409258185ec6d604dbd4fbd5d459b3\"><img alt=\"Fig13 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/83\/Fig13_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 13<\/b> Example of the prescribing dashboard the teams would add functionally to following the Scrum framework.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The first part of this unit involved teaching fundamental Python and databases skills using SQL to students, many of whom have had no or limited exposure to computer programming (coding). We implemented the teaching of Python and SQL in the Jupyter Notebook environment. The unit was a module available as part of the master's module on a health informatics program that is a joint award between the University of Manchester and University College London (UCL).<sup id=\"rdp-ebb-cite_ref-UMMsCPGDip_25-0\" class=\"reference\"><a href=\"#cite_note-UMMsCPGDip-25\">[25]<\/a><\/sup> The principal challenge faced was delivering teaching of coding skills to those who have little or no coding experience via distance learning in a way that allowed them to focus on obtaining these fundamental skills in the chosen language (Python) without introducing any additional complexity to the process. A further challenge was that unlike undergraduate courses where we may teach in person using a PC cluster (computer room\/lab) with preloaded software managed by IT services, many master\u2019s level students will be required (and usually prefer) to use their own computing devices (e.g., laptops, tablets, desktops). Supporting the use of software on these different operating systems and platforms adds an additional challenge. In order to remove or reduce these barriers to learning, we decided to make use of the interactive Jupyter Notebooks that support among others the Python programming language. We were then able to host a set of notebooks taking students through the various coding topics in order (Figure 14). A link to the notebooks was provided on the VLE for the module (i.e., Moodle) and the universities central username and password system added to prevent non-university affiliated personnel accessing the notebooks.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig14_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"1721a65c5ad5154127bb09490cdbd4ac\"><img alt=\"Fig14 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d6\/Fig14_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 14<\/b> List of notebooks covering the various topics of programming with Python.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The main reason we built our own notebooks rather than link to other existing resources (e.g., <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/software-carpentry.org\/\" target=\"_blank\">Software Carpentry<\/a>) was to provide specific health-related examples for the students so that the domain would be familiar to them. Many of the computer science examples can be abstract in nature. By providing concrete health examples, it was hoped that this would help the students to see the relevance of potential applications of programming in health care settings. We were also able to add tasks throughout the notebooks that allowed students to code in the notebook and then view a model answer (e.g., Figure 15) using the exercise extension discussed previously.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig15_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"35684a9f20cd6897fb6c6b2a46bc5f5d\"><img alt=\"Fig15 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/ae\/Fig15_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 15<\/b> Example of task from notebook. Clicking the \u201cShow Solution\u201d button reveals the model answer.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Figure 16 shows an example of a notebook from the set about the topic of variables and strings.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig16_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"a1a84bde8ef7bbaa9d6b3fee5c4757a1\"><img alt=\"Fig16 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/37\/Fig16_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 16<\/b> Example of notebook on variables and strings programming topics.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Using the notebooks in this way allowed us to sidestep the issues of asking the students to download and set up Python on their machines with the associated complexities of supporting this. We do this later on in the unit where we move to group work and using an integrated development environment (IDE). At the beginning of the unit, we remove this barrier and allow the students to focus on learning the Python language and programming fundamentals. Initial feedback suggests that this improved their confidence with coding prior to the summative portion of the module. We provided support for students using the notebooks through the VLE and also by Slack (a cloud-based instant messaging service). Teaching assistants (TAs) would monitor the Slack channels and respond to issues the students faced with running the notebooks and Python in general.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Case_study_2:_Introduction_to_Programming\">Case study 2: Introduction to Programming<\/span><\/h3>\n<p>A second case study involving the teaching of basic coding in Python was a 15-credit module in a new distance learning PG Cert in clinical bioinformatics. It was designed to teach the fundamentals of <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomics<\/a> medicine to a diverse cohort of students. Clinical bioinformatics is a relatively new profession and represents the marriage of computer science with clinical practice. The computational and data skills needed to become a clinical bioinformatician are in short supply in the NHS, with training and education trying to fill the skills gap.<sup id=\"rdp-ebb-cite_ref-AttwoodAGlob19_26-0\" class=\"reference\"><a href=\"#cite_note-AttwoodAGlob19-26\">[26]<\/a><\/sup> Those new to the field could come from many backgrounds, for example, those from the health sector with little or no programming experience to those with IT knowledge but with limited clinical experience.\n<\/p><p>Similarly, to case study 1, the module also adhered to agile principles but was delivered entirely online. The first part of the unit involved teaching basic GitHub and Python skills to students with differing levels of programming experience. It needed to support these varied learning requirements but also support students remotely, without face-to-face contact, while emulating clinical bioinformatics in practice. We therefore created an immersive and realistic software development environment with real-world practice-based problems in the form of sprints. To ensure an authentic learning experience, the students were taught to use Anaconda to install Python 3 onto their own machines. They also installed Git, and Windows users also installed and initialised Git Bash so that all students could be taught in a Linux environment. The course content was delivered to the students outside of Blackboard (learning management system) using <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/i3hsInnovation\" target=\"_blank\">GitHub<\/a>.\n<\/p><p>Other than initial introductory materials, the course material was taught using Jupyter Notebooks. The notebooks allowed us to provide interactive teaching on the basic principles of Python programming, including exercises that the students could complete within the notebook to hone their skills. Once the basic principles of Python programming were covered, we introduced the students to representational state transfer (REST) <a href=\"https:\/\/www.limswiki.org\/index.php\/Application_programming_interface\" title=\"Application programming interface\" class=\"wiki-link\" data-key=\"36fc319869eba4613cb0854b421b0934\">application programming interfaces<\/a> (APIs) commonly used to collate genomics data. The immersive nature of the notebooks allowed us to build authentic tutorials to help students understand how data are retrieved from REST APIs and how they could build their own REST APIs. The notebooks gave the students the space to practice and develop these new skills comfortably in a fail-safe environment while using real-world examples. The flexibility of the notebooks also meant we could reuse them easily and incorporate slightly different examples to support the diverse student cohort.\n<\/p><p>The notebooks introduced the team-based sprint scenarios requiring the students to prototype code that will meet real-world needs of NHS scientists, as well as an in-production genomics software application, VariantValidator.<sup id=\"rdp-ebb-cite_ref-FreemanVariant18_27-0\" class=\"reference\"><a href=\"#cite_note-FreemanVariant18-27\">[27]<\/a><\/sup> The interactive and engaging teaching provided by the notebooks helped scaffold the learning with short snippets of interactive code. These blocks of learning eventually culminated in a final SPRINT project where the learners built resources based on needs from their own practice (or became additional prototypes to support the VariantValidator project).\n<\/p><p>Other tools such as Slack helped with the group work and educational support, such as solving initial configuration issues, pastoral support, and providing personal feedback on SPRINT activities. This peer-supported learning approach helped hone another essential skill in clinical bioinformatics, like being an active member of a community of practice.<sup id=\"rdp-ebb-cite_ref-DaviesProblem19_28-0\" class=\"reference\"><a href=\"#cite_note-DaviesProblem19-28\">[28]<\/a><\/sup> It was the dual approach of active learning materials providing a fail-safe environment in the notebooks, coupled with the peer-supported learning via Slack; this meant we were able to deliver effective training into multiple countries, including a student working in frontline healthcare in China during the peak of the <a href=\"https:\/\/www.limswiki.org\/index.php\/COVID-19\" title=\"COVID-19\" class=\"mw-redirect wiki-link\" data-key=\"da9bd20c492b2a17074ad66c2fe25652\">COVID-19<\/a> . At the end of the course, because the notebooks were downloaded to the students\u2019 machines, they had the tools, tutorials, and examples at their fingertips to learn back in practice. The aim of this \u201csandbox\u201d of editable and authentic learning materials was to help students to strengthen their programming skills in the long term and progress as members of the wider clinical bioinformatics community.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Evaluation\">Evaluation<\/span><\/h3>\n<p>A detailed evaluation is beyond the scope of this paper as we are yet to run the various modules for significant time to collect sufficient data. We do, however, present some initial findings from a survey carried out on units using Jupyter Notebooks for teaching, as well as some statements from students about their experiences. Twelve students completed the survey and were asked six questions concerning the use of Jupyter Notebooks. Those six questions were:\n<\/p>\n<ol><li> How useful did you find this course unit? (1 = not at all, 10 = very useful)<\/li>\n<li> How easy was it to use the Jupyter Notebooks in your learning? (1 = very difficult, 10 = very easy)<\/li>\n<li> Did the notebooks structure and combination of activities help you build understanding? (no\/yes)<\/li>\n<li> Did the pace of activities feel right to you? (no\/yes)<\/li>\n<li> How likely would you be to recommend Jupyter Notebooks and the learning approach we have followed? (1 = not at all likely, 10 = extremely likely)<\/li>\n<li> Overall, how satisfied were you with the course? (1 = least happy, 10 = happiest)<\/li><\/ol>\n<p>The results can be seen in Fig 17. We found that students provided predominantly positive responses to the questions asked. Results show that the students indicated that they would recommend notebooks for learning, found the course unit useful, and were satisfied with the course. For case study 2, students also provided reflective videos and feedback. This included the following comments on the practice-focused Jupyter Notebooks:\n<\/p>\n<blockquote>\"[\u2026] the programming module starts with the basics for students where it is new to them. It gives an excellent overview of the different methodologies and languages and resources that are key to bioinformatics and what\u2019s also really helpful, or I found helpful, is that the code is taught in snippets in Juptyer Notebooks so you are able to try out small parts of the code for yourself \u2026 before you even need to get to grips with the development environment. So that was really useful.\"\n<br \/><i>\u2014student on PG Cert clinical bioinformatics in their <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/youtu.be\/TiIEyEeNiaU\" target=\"_blank\">video feedback<\/a> of the course (at 2 minutes 15 seconds)<\/i><\/blockquote>\n<blockquote>\u201cAs an NHS clinician with very little experience of coding, the course and specifically the introduction to programming has a steep learning curve. The modules have all been challenging but the accessibility of tutor support and their proactive approach to supporting students has meant that I\u2019ve never felt lost. As a non-specialist in this field, the course has provided me with the toolkit to understand the specific role that bioinformatics plays within the NHS. Whether one goes on undertake further study in this field or not, this PGCert course covers much of the material that a clinician will need familiarity with in the evolving healthcare landscape.\u201d<\/blockquote>\n<blockquote>\u201c[\u2026] very grateful for the quality of teaching on the course (across all the modules).\u201d\n<br \/><i>\u2014student on PG Cert clinical bioinformatics, who created a <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.youtube.com\/watch?v=F-YwweY2K-4&feature=youtu.be&hd=1\" target=\"_blank\">reflective presentation<\/a> on Introduction to Programming<\/i><\/blockquote>\n<p>(Permission was obtained from students to use their statements and videos in publications and for marketing purposes. Videos are publically available on YouTube.)\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig17_Davies_PLOSCompBio20_16-11.png\" class=\"image wiki-link\" data-key=\"cf49aa4e382a4c541acc7445c40bceca\"><img alt=\"Fig17 Davies PLOSCompBio20 16-11.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3b\/Fig17_Davies_PLOSCompBio20_16-11.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 17<\/b> Results of notebook student survey (<i>n<\/i> = 12).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>One has to be cautious when introducing technology like Jupyter Notebook into the classroom, especially when running a distance learning or online course if a large part of the course unit is dependent on notebook content. Such notebooks should be tested thoroughly and technical support available for their maintenance and any issues that may arise. Their use may also be more or less problematic with different user groups. Although it is more likely that those from a science, technology, engineering and math (STEM) background will be more comfortable with such tools, we cannot assume that this is necessarily the case. Myths like that of the \u201cdigital native\u201d (those born in the age of pervasive digital media) having some special advantages over other generations have been proven to be an unhelpful stereotype.<sup id=\"rdp-ebb-cite_ref-KirschnerTheMyths17_29-0\" class=\"reference\"><a href=\"#cite_note-KirschnerTheMyths17-29\">[29]<\/a><\/sup> This means that one has to provide adequate support for the use of such tools to ensure their smooth adoption catering for different levels of digital literacy. Support for students with accessibility needs is also a consideration, and where possible, web content should conform to Web Content Accessibility Guidelines (WCAG).<sup id=\"rdp-ebb-cite_ref-KirkpatrickWeb18_30-0\" class=\"reference\"><a href=\"#cite_note-KirkpatrickWeb18-30\">[30]<\/a><\/sup> To achieve these aims, it is useful to place the students at the center of the design process, considering who the target audience is and their needs, the application of the desired learning principles, how they will be presented, and importantly, how this design stands the test of time and is able to be adapted to meet changing needs.<sup id=\"rdp-ebb-cite_ref-BeethamReth13_31-0\" class=\"reference\"><a href=\"#cite_note-BeethamReth13-31\">[31]<\/a><\/sup>\n<\/p><p>For the MIE module (case study 1), we were careful not to assume prior knowledge, especially given the diverse nature of the backgrounds and experience of health informatics students enrolled on that module. As this was applied to a blended module, we could not make use of standard computer clusters with preloaded software. Most of the students would be accessing the module using their own computing devices; therefore, we wanted to avoid the setup issues of downloading and installing a Python distribution (at least initially) until they had gained some familiarity and confidence with coding. We also didn\u2019t want to introduce an IDE at the initial stage of the module or use the console, as this is not ideal for writing larger blocks of code. These issues were overcome by remotely hosting the notebooks using cloud services and providing a link for the students to log in via the main university login system. This way, they would have their own secure copy of the notebooks for the module that could be accessed and modified, allowing them to rapidly focus on writing and learning to code, rather than all of the peripheral setup requirements and support issues.\n<\/p><p>In contrast, the approach for the Introduction to Programming module was to provide an authentic and immersive learning journey. The aim was to try to simulate everyday clinical bioinformatics working practices but in a safe learning environment that could give them the space to fail and learn from their mistakes both individually and as a team. This meant students needed to download the notebooks locally, work with different versions using GitHub, and work as a team on Slack. The challenge was to provide enough support to help deal with any issues they had with the tools and techniques being taught, but with enough autonomy so they could develop much-needed problem-solving skills in clinical bioinformatics. This balancing act required a lot of resource both at the design stage, with additional materials for different learner requirements, and during the delivery stage.\n<\/p><p>Supporting modules such as these may require more resources and support than traditional face-to-face modalities. The interactive coding tasks helped the student to gain hands-on experience in coding while customising their own class notes, which they could download and keep beyond the duration of the course. Such tasks and interactive elements provided via notebooks uniquely helped to move students from a more static learning experience into a more dynamic experience.<sup id=\"rdp-ebb-cite_ref-BarbaTeach19_17-1\" class=\"reference\"><a href=\"#cite_note-BarbaTeach19-17\">[17]<\/a><\/sup> This can provide a deeper level of immersion in the tasks (for example, exploring a dataset via an interactive plot or applying skills to a practice-based problem of their choosing). In terms of Bloom's (revised) taxonomy, this moves students towards the top of higher ordered thinking into creation and production.<sup id=\"rdp-ebb-cite_ref-AndersonTax01_32-0\" class=\"reference\"><a href=\"#cite_note-AndersonTax01-32\">[32]<\/a><\/sup>\n<\/p><p>Careful consideration should also be afforded to the overall aims of the module, unit, or course, and technology should be used where appropriate to improve or facilitate learning, rather than be used for novelty purposes. Findings suggest that learning should be the main focus, rather than an aim to be \u201ctech-centric.\u201d<sup id=\"rdp-ebb-cite_ref-CurryPutting18_33-0\" class=\"reference\"><a href=\"#cite_note-CurryPutting18-33\">[33]<\/a><\/sup> If we want to apply digital pedagogy successfully, we need to match each bit of technology with our required pedagogical goals.<sup id=\"rdp-ebb-cite_ref-CurryPutting18_33-1\" class=\"reference\"><a href=\"#cite_note-CurryPutting18-33\">[33]<\/a><\/sup> Essentially, the technology is there to enhance the learning and should be chosen to support the fulfilment of the learning objectives while considering the students\u2019 wider contexts and learning environments. A further consideration is the quality and practices that we impart through such methods. There are calls for journal editors and reviewers to enforce computational reproducibility<sup id=\"rdp-ebb-cite_ref-GymrekRecomm16_34-0\" class=\"reference\"><a href=\"#cite_note-GymrekRecomm16-34\">[34]<\/a><\/sup>; however, while many scientists use and write code on a regular basis, they often lack formal training in good software engineering practices.<sup id=\"rdp-ebb-cite_ref-WilsonBest14_35-0\" class=\"reference\"><a href=\"#cite_note-WilsonBest14-35\">[35]<\/a><\/sup> To embed good practices that students can use in their further research careers, we need to ensure that whenever possible, the content we generate helps to distill these practices. This in turn negates the importance of providing appropriate training for educators themselves.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The use of Jupyter Notebooks on several of our university modules has been positively received by both staff and students, who see them as a useful resource for learning to code and communicate research findings and analysis, and in the cases presented, learning the Python programming language specifically. The use of notebooks in such units also gives students an introduction to the notebook environment, which some may go on to use for research purposes later in their career or for the research component of their masters degrees. The use of digital notebooks and other technologies should be carefully evaluated to ensure they add real value to the learning aims and objectives, placing the pedagogic aims of the course at the centre of the process. Given that the use of such tools is becoming more ubiquitous in the bioscience research and scientific education domains, it would be advantageous for academic tutors in such fields to have an awareness and understanding of their application and to consider their use for providing interactive components to computational learning tasks where appropriate.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>There was no funding for this project.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors have declared that no competing interests exist.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-GregoryProf13-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GregoryProf13_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gregory, J.; Salmon, G. (2013). \"Professional development for online university teaching\". <i>Distance Education<\/i> <b>34<\/b> (3): 256\u201370. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F01587919.2013.835771\" target=\"_blank\">10.1080\/01587919.2013.835771<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Professional+development+for+online+university+teaching&rft.jtitle=Distance+Education&rft.aulast=Gregory%2C+J.%3B+Salmon%2C+G.&rft.au=Gregory%2C+J.%3B+Salmon%2C+G.&rft.date=2013&rft.volume=34&rft.issue=3&rft.pages=256%E2%80%9370&rft_id=info:doi\/10.1080%2F01587919.2013.835771&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeorginaInteg08-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeorginaInteg08_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Georgina, D.A.; Olson, M.R. (2008). \"Integration of technology in higher education: A review of faculty self-perceptions\". <i>The Internet and Higher Education<\/i> <b>11<\/b> (1): 1\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.iheduc.2007.11.002\" target=\"_blank\">10.1016\/j.iheduc.2007.11.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integration+of+technology+in+higher+education%3A+A+review+of+faculty+self-perceptions&rft.jtitle=The+Internet+and+Higher+Education&rft.aulast=Georgina%2C+D.A.%3B+Olson%2C+M.R.&rft.au=Georgina%2C+D.A.%3B+Olson%2C+M.R.&rft.date=2008&rft.volume=11&rft.issue=1&rft.pages=1%E2%80%938&rft_id=info:doi\/10.1016%2Fj.iheduc.2007.11.002&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JupyterHome-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JupyterHome_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-JupyterHome_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/jupyter.org\/\" target=\"_blank\">\"Jupyter\"<\/a>. Project Jupyter<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/jupyter.org\/\" target=\"_blank\">https:\/\/jupyter.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Jupyter&rft.atitle=&rft.pub=Project+Jupyter&rft_id=https%3A%2F%2Fjupyter.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RichardsonScien18-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RichardsonScien18_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-RichardsonScien18_4-1\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Richardson, M.L.; Amini, B.. \"Scientific Notebook Software: Applications for Academic Radiology\". <i>Current Problems in Diagnostic Radiology<\/i> <b>47<\/b> (6): 368\u201377. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1067%2Fj.cpradiol.2017.09.005\" target=\"_blank\">10.1067\/j.cpradiol.2017.09.005<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29122394\" target=\"_blank\">29122394<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Scientific+Notebook+Software%3A+Applications+for+Academic+Radiology&rft.jtitle=Current+Problems+in+Diagnostic+Radiology&rft.aulast=Richardson%2C+M.L.%3B+Amini%2C+B.&rft.au=Richardson%2C+M.L.%3B+Amini%2C+B.&rft.volume=47&rft.issue=6&rft.pages=368%E2%80%9377&rft_id=info:doi\/10.1067%2Fj.cpradiol.2017.09.005&rft_id=info:pmid\/29122394&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MGMarkdown-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MGMarkdown_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Cone, M.. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.markdownguide.org\/\" target=\"_blank\">\"Markdown Guide\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.markdownguide.org\/\" target=\"_blank\">https:\/\/www.markdownguide.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 September 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Markdown+Guide&rft.atitle=&rft.aulast=Cone%2C+M.&rft.au=Cone%2C+M.&rft_id=https%3A%2F%2Fwww.markdownguide.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LatexAbout-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LatexAbout_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.latex-project.org\/about\/\" target=\"_blank\">\"An introduction to LaTeX\"<\/a>. The LaTeX Project<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.latex-project.org\/about\/\" target=\"_blank\">https:\/\/www.latex-project.org\/about\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=An+introduction+to+LaTeX&rft.atitle=&rft.pub=The+LaTeX+Project&rft_id=https%3A%2F%2Fwww.latex-project.org%2Fabout%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WBLaTeX-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WBLaTeX_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Roberts, A.; Oetiker, T.; Partl, H. et al.. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"https:\/\/en.wikibooks.org\/wiki\/LaTeX\/Mathematics\" data-key=\"997d0b17fb2c0b340ee145cbe5c00d48\">\"LaTeX\/Mathematics\"<\/a>. <i>LaTeX - WikiBooks<\/i><span class=\"printonly\">. <a rel=\"nofollow\" class=\"external free wiki-link\" href=\"https:\/\/en.wikibooks.org\/wiki\/LaTeX\/Mathematics\" data-key=\"997d0b17fb2c0b340ee145cbe5c00d48\">https:\/\/en.wikibooks.org\/wiki\/LaTeX\/Mathematics<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=LaTeX%2FMathematics&rft.atitle=LaTeX+-+WikiBooks&rft.aulast=Roberts%2C+A.%3B+Oetiker%2C+T.%3B+Partl%2C+H.+et+al.&rft.au=Roberts%2C+A.%3B+Oetiker%2C+T.%3B+Partl%2C+H.+et+al.&rft_id=https%3A%2F%2Fen.wikibooks.org%2Fwiki%2FLaTeX%2FMathematics&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AnacondaDist-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AnacondaDist_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20190906215334\/https:\/\/www.anaconda.com\/distribution\/\" target=\"_blank\">\"Anaconda Distribution\"<\/a>. Anaconda, Inc. 2019. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.anaconda.com\/distribution\/\" target=\"_blank\">the original<\/a> on 06 September 2019<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20190906215334\/https:\/\/www.anaconda.com\/distribution\/\" target=\"_blank\">https:\/\/web.archive.org\/web\/20190906215334\/https:\/\/www.anaconda.com\/distribution\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 September 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Anaconda+Distribution&rft.atitle=&rft.date=2019&rft.pub=Anaconda%2C+Inc&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20190906215334%2Fhttps%3A%2F%2Fwww.anaconda.com%2Fdistribution%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JupyterHub-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JupyterHub_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/jupyter.org\/hub\" target=\"_blank\">\"JupyterHub\"<\/a>. Project Jupyter<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/jupyter.org\/hub\" target=\"_blank\">https:\/\/jupyter.org\/hub<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=JupyterHub&rft.atitle=&rft.pub=Project+Jupyter&rft_id=https%3A%2F%2Fjupyter.org%2Fhub&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PJUsing-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PJUsing_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ipywidgets.readthedocs.io\/en\/latest\/examples\/Using%20Interact.html\" target=\"_blank\">\"Using Interact\"<\/a>. <i>ipywidgets User Guide<\/i>. Project Jupyter<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ipywidgets.readthedocs.io\/en\/latest\/examples\/Using%20Interact.html\" target=\"_blank\">https:\/\/ipywidgets.readthedocs.io\/en\/latest\/examples\/Using%20Interact.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Using+Interact&rft.atitle=ipywidgets+User+Guide&rft.pub=Project+Jupyter&rft_id=https%3A%2F%2Fipywidgets.readthedocs.io%2Fen%2Flatest%2Fexamples%2FUsing%2520Interact.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WTWhatRes20-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WTWhatRes20_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/wellcome.org\/reports\/what-researchers-think-about-research-culture\" target=\"_blank\">\"What researchers think about the culture they work in\"<\/a>. Wellcome Trust. 14 January 2020<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/wellcome.org\/reports\/what-researchers-think-about-research-culture\" target=\"_blank\">https:\/\/wellcome.org\/reports\/what-researchers-think-about-research-culture<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+researchers+think+about+the+culture+they+work+in&rft.atitle=&rft.date=14+January+2020&rft.pub=Wellcome+Trust&rft_id=https%3A%2F%2Fwellcome.org%2Freports%2Fwhat-researchers-think-about-research-culture&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HardwickeData18-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HardwickeData18_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hardwicke, T.E.; Mathur, M.B.; MacDonald, K. et al. (2018). \"Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal <i>Cognition<\/i>\". <i>Royal Society Open Science<\/i> <b>5<\/b> (8): 180448. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1098%2Frsos.180448\" target=\"_blank\">10.1098\/rsos.180448<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+availability%2C+reusability%2C+and+analytic+reproducibility%3A+Evaluating+the+impact+of+a+mandatory+open+data+policy+at+the+journal+%27%27Cognition%27%27&rft.jtitle=Royal+Society+Open+Science&rft.aulast=Hardwicke%2C+T.E.%3B+Mathur%2C+M.B.%3B+MacDonald%2C+K.+et+al.&rft.au=Hardwicke%2C+T.E.%3B+Mathur%2C+M.B.%3B+MacDonald%2C+K.+et+al.&rft.date=2018&rft.volume=5&rft.issue=8&rft.pages=180448&rft_id=info:doi\/10.1098%2Frsos.180448&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UKRNHome-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UKRNHome_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ukrn.org\/\" target=\"_blank\">\"U.K. Reproducibility Network\"<\/a>. U.K. Reproducibility Network<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ukrn.org\/\" target=\"_blank\">https:\/\/www.ukrn.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=U.K.+Reproducibility+Network&rft.atitle=&rft.pub=U.K.+Reproducibility+Network&rft_id=https%3A%2F%2Fwww.ukrn.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Munaf.C3.B2Raising19-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Munaf.C3.B2Raising19_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Munaf\u00f2, M. (2019). \"Raising research quality will require collective action\". <i>Nature<\/i> <b>576<\/b>: 183. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fd41586-019-03750-7\" target=\"_blank\">10.1038\/d41586-019-03750-7<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Raising+research+quality+will+require+collective+action&rft.jtitle=Nature&rft.aulast=Munaf%C3%B2%2C+M.&rft.au=Munaf%C3%B2%2C+M.&rft.date=2019&rft.volume=576&rft.pages=183&rft_id=info:doi\/10.1038%2Fd41586-019-03750-7&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RuleTen18-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RuleTen18_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rule, A.; Birmingham, A.; Zuniga, C. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/arxiv.org\/abs\/1810.08055\" target=\"_blank\">\"Ten Simple Rules for Reproducible Research in Jupyter Notebooks\"<\/a>. <i>arXiv<\/i>: 1\u20138<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/arxiv.org\/abs\/1810.08055\" target=\"_blank\">https:\/\/arxiv.org\/abs\/1810.08055<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ten+Simple+Rules+for+Reproducible+Research+in+Jupyter+Notebooks&rft.jtitle=arXiv&rft.aulast=Rule%2C+A.%3B+Birmingham%2C+A.%3B+Zuniga%2C+C.+et+al.&rft.au=Rule%2C+A.%3B+Birmingham%2C+A.%3B+Zuniga%2C+C.+et+al.&rft.date=2018&rft.pages=1%E2%80%938&rft_id=https%3A%2F%2Farxiv.org%2Fabs%2F1810.08055&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MedleyTellur18-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MedleyTellur18_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Medley, J.K.; Choi, K.; K\u00f6nig, M. et al. (2018). \"Tellurium notebooks\u2014An environment for reproducible dynamical modeling in systems biology\". <i>PLOS Computational Biology<\/i> <b>14<\/b> (6): e1006220. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1006220\" target=\"_blank\">10.1371\/journal.pcbi.1006220<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Tellurium+notebooks%E2%80%94An+environment+for+reproducible+dynamical+modeling+in+systems+biology&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Medley%2C+J.K.%3B+Choi%2C+K.%3B+K%C3%B6nig%2C+M.+et+al.&rft.au=Medley%2C+J.K.%3B+Choi%2C+K.%3B+K%C3%B6nig%2C+M.+et+al.&rft.date=2018&rft.volume=14&rft.issue=6&rft.pages=e1006220&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1006220&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarbaTeach19-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BarbaTeach19_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-BarbaTeach19_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Barba, L.A.; Barker, L.J.; Blank, D.S. et al. (06 December 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/jupyter4edu.github.io\/jupyter-edu-book\/index.html\" target=\"_blank\">\"Teaching and Learning with Jupyter\"<\/a>. GitHub<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/jupyter4edu.github.io\/jupyter-edu-book\/index.html\" target=\"_blank\">https:\/\/jupyter4edu.github.io\/jupyter-edu-book\/index.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Teaching+and+Learning+with+Jupyter&rft.atitle=&rft.aulast=Barba%2C+L.A.%3B+Barker%2C+L.J.%3B+Blank%2C+D.S.+et+al.&rft.au=Barba%2C+L.A.%3B+Barker%2C+L.J.%3B+Blank%2C+D.S.+et+al.&rft.date=06+December+2019&rft.pub=GitHub&rft_id=https%3A%2F%2Fjupyter4edu.github.io%2Fjupyter-edu-book%2Findex.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-nbgrader-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-nbgrader_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/nbgrader.readthedocs.io\/en\/stable\/\" target=\"_blank\">\"nbgrader\"<\/a>. Jupyter Development Team<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/nbgrader.readthedocs.io\/en\/stable\/\" target=\"_blank\">https:\/\/nbgrader.readthedocs.io\/en\/stable\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=nbgrader&rft.atitle=&rft.pub=Jupyter+Development+Team&rft_id=https%3A%2F%2Fnbgrader.readthedocs.io%2Fen%2Fstable%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Binder-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Binder_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/mybinder.org\/\" target=\"_blank\">\"binder\"<\/a>. Project Jupyter<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/mybinder.org\/\" target=\"_blank\">https:\/\/mybinder.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 June 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=binder&rft.atitle=&rft.pub=Project+Jupyter&rft_id=https%3A%2F%2Fmybinder.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BussonnierBinder18-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BussonnierBinder18_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-BussonnierBinder18_20-1\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bussonnier, M.; Forde, J.; Freeman, J. et al. (2018). \"Binder 2.0 - Reproducible, interactive, sharable environments for science at scale\". <i>Proceedings of the 17th Python in Science Conference<\/i>: 113\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.25080%2FMajora-4af1f417-011\" target=\"_blank\">10.25080\/Majora-4af1f417-011<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Binder+2.0+-+Reproducible%2C+interactive%2C+sharable+environments+for+science+at+scale&rft.jtitle=Proceedings+of+the+17th+Python+in+Science+Conference&rft.aulast=Bussonnier%2C+M.%3B+Forde%2C+J.%3B+Freeman%2C+J.+et+al.&rft.au=Bussonnier%2C+M.%3B+Forde%2C+J.%3B+Freeman%2C+J.+et+al.&rft.date=2018&rft.pages=113%E2%80%9320&rft_id=info:doi\/10.25080%2FMajora-4af1f417-011&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DCSharing-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DCSharing_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Data Carpentry. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/reproducible-science-curriculum.github.io\/sharing-RR-Jupyter\/\" target=\"_blank\">\"Sharing Jupyter Notebooks\"<\/a>. GitHub<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/reproducible-science-curriculum.github.io\/sharing-RR-Jupyter\/\" target=\"_blank\">https:\/\/reproducible-science-curriculum.github.io\/sharing-RR-Jupyter\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 June 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Sharing+Jupyter+Notebooks&rft.atitle=&rft.aulast=Data+Carpentry&rft.au=Data+Carpentry&rft.pub=GitHub&rft_id=https%3A%2F%2Freproducible-science-curriculum.github.io%2Fsharing-RR-Jupyter%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScrumWhatIs-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScrumWhatIs_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.scrum.org\/resources\/what-is-scrum\" target=\"_blank\">\"What Is Scrum?\"<\/a>. Advanced Development Methods, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.scrum.org\/resources\/what-is-scrum\" target=\"_blank\">https:\/\/www.scrum.org\/resources\/what-is-scrum<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 25 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+Is+Scrum%3F&rft.atitle=&rft.pub=Advanced+Development+Methods%2C+Inc&rft_id=https%3A%2F%2Fwww.scrum.org%2Fresources%2Fwhat-is-scrum&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Agile101-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Agile101_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.agilealliance.org\/agile101\/\" target=\"_blank\">\"Agile 101\"<\/a>. Agile Alliance<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.agilealliance.org\/agile101\/\" target=\"_blank\">https:\/\/www.agilealliance.org\/agile101\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 25 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Agile+101&rft.atitle=&rft.pub=Agile+Alliance&rft_id=https%3A%2F%2Fwww.agilealliance.org%2Fagile101%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FlaskDoc-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FlaskDoc_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/flask.palletsprojects.com\/en\/1.1.x\/\" target=\"_blank\">\"Flask Documentation\"<\/a>. Pallets<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/flask.palletsprojects.com\/en\/1.1.x\/\" target=\"_blank\">https:\/\/flask.palletsprojects.com\/en\/1.1.x\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Flask+Documentation&rft.atitle=&rft.pub=Pallets&rft_id=https%3A%2F%2Fflask.palletsprojects.com%2Fen%2F1.1.x%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UMMsCPGDip-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UMMsCPGDip_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.manchester.ac.uk\/study\/masters\/courses\/list\/12478\/msc-pgdip-pgcert-health-informatics-ucl-uom-joint-award\/\" target=\"_blank\">\"MSc\/PGDip\/PGCert Health Informatics (UCL\/UoM Joint Award) \/ Overview\"<\/a>. University of Manchester<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.manchester.ac.uk\/study\/masters\/courses\/list\/12478\/msc-pgdip-pgcert-health-informatics-ucl-uom-joint-award\/\" target=\"_blank\">https:\/\/www.manchester.ac.uk\/study\/masters\/courses\/list\/12478\/msc-pgdip-pgcert-health-informatics-ucl-uom-joint-award\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MSc%2FPGDip%2FPGCert+Health+Informatics+%28UCL%2FUoM+Joint+Award%29+%2F+Overview&rft.atitle=&rft.pub=University+of+Manchester&rft_id=https%3A%2F%2Fwww.manchester.ac.uk%2Fstudy%2Fmasters%2Fcourses%2Flist%2F12478%2Fmsc-pgdip-pgcert-health-informatics-ucl-uom-joint-award%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AttwoodAGlob19-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AttwoodAGlob19_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Attwood, T.K.; Blackford, S.; Brazas, M.D. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6433731\" target=\"_blank\">\"A global perspective on evolving bioinformatics and data science training needs\"<\/a>. <i>Briefings in Bioinformatics<\/i> <b>20<\/b> (2): 398\u2013404. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbx100\" target=\"_blank\">10.1093\/bib\/bbx100<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6433731\/\" target=\"_blank\">PMC6433731<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28968751\" target=\"_blank\">28968751<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6433731\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6433731<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+global+perspective+on+evolving+bioinformatics+and+data+science+training+needs&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Attwood%2C+T.K.%3B+Blackford%2C+S.%3B+Brazas%2C+M.D.+et+al.&rft.au=Attwood%2C+T.K.%3B+Blackford%2C+S.%3B+Brazas%2C+M.D.+et+al.&rft.date=2019&rft.volume=20&rft.issue=2&rft.pages=398%E2%80%93404&rft_id=info:doi\/10.1093%2Fbib%2Fbbx100&rft_id=info:pmc\/PMC6433731&rft_id=info:pmid\/28968751&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6433731&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FreemanVariant18-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FreemanVariant18_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Freeman, P.J.; Hart, R.K.; Gretton, L.J. et al. (2018). \"VariantValidator: Accurate validation, mapping, and formatting of sequence variation descriptions\". <i>Human Mutation<\/i> <b>39<\/b> (1): 61\u201368. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fhumu.23348\" target=\"_blank\">10.1002\/humu.23348<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=VariantValidator%3A+Accurate+validation%2C+mapping%2C+and+formatting+of+sequence+variation+descriptions&rft.jtitle=Human+Mutation&rft.aulast=Freeman%2C+P.J.%3B+Hart%2C+R.K.%3B+Gretton%2C+L.J.+et+al.&rft.au=Freeman%2C+P.J.%3B+Hart%2C+R.K.%3B+Gretton%2C+L.J.+et+al.&rft.date=2018&rft.volume=39&rft.issue=1&rft.pages=61%E2%80%9368&rft_id=info:doi\/10.1002%2Fhumu.23348&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DaviesProblem19-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DaviesProblem19_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Davies, A.C.; Harris, D.; Banks-Gatenby, A. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6597031\" target=\"_blank\">\"Problem-based learning in clinical bioinformatics education: Does it help to create communities of practice?\"<\/a>. <i>PLOS Computational Biology<\/i> <b>15<\/b> (6): e1006746. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1006746\" target=\"_blank\">10.1371\/journal.pcbi.1006746<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6597031\/\" target=\"_blank\">PMC6597031<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31246944\" target=\"_blank\">31246944<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6597031\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6597031<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Problem-based+learning+in+clinical+bioinformatics+education%3A+Does+it+help+to+create+communities+of+practice%3F&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Davies%2C+A.C.%3B+Harris%2C+D.%3B+Banks-Gatenby%2C+A.+et+al.&rft.au=Davies%2C+A.C.%3B+Harris%2C+D.%3B+Banks-Gatenby%2C+A.+et+al.&rft.date=2019&rft.volume=15&rft.issue=6&rft.pages=e1006746&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1006746&rft_id=info:pmc\/PMC6597031&rft_id=info:pmid\/31246944&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6597031&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KirschnerTheMyths17-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KirschnerTheMyths17_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kirschner, P.A.; De Bruyckere, P. (2017). \"The myths of the digital native and the multitasker\". <i>Teaching and Teacher Education<\/i> <b>67<\/b>: 135\u201342. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.tate.2017.06.001\" target=\"_blank\">10.1016\/j.tate.2017.06.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+myths+of+the+digital+native+and+the+multitasker&rft.jtitle=Teaching+and+Teacher+Education&rft.aulast=Kirschner%2C+P.A.%3B+De+Bruyckere%2C+P.&rft.au=Kirschner%2C+P.A.%3B+De+Bruyckere%2C+P.&rft.date=2017&rft.volume=67&rft.pages=135%E2%80%9342&rft_id=info:doi\/10.1016%2Fj.tate.2017.06.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KirkpatrickWeb18-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KirkpatrickWeb18_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Kirkpatrick, A.; O'Connor, J.O.; Campbell, A. et al. (05 June 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.w3.org\/TR\/WCAG21\/\" target=\"_blank\">\"Web Content Accessibility Guidelines (WCAG) 2.1\"<\/a>. W3C<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.w3.org\/TR\/WCAG21\/\" target=\"_blank\">https:\/\/www.w3.org\/TR\/WCAG21\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 25 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Web+Content+Accessibility+Guidelines+%28WCAG%29+2.1&rft.atitle=&rft.aulast=Kirkpatrick%2C+A.%3B+O%27Connor%2C+J.O.%3B+Campbell%2C+A.+et+al.&rft.au=Kirkpatrick%2C+A.%3B+O%27Connor%2C+J.O.%3B+Campbell%2C+A.+et+al.&rft.date=05+June+2018&rft.pub=W3C&rft_id=https%3A%2F%2Fwww.w3.org%2FTR%2FWCAG21%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BeethamReth13-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BeethamReth13_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Beetham, H.; Sharpe, R., ed. (2013). <i>Rethinking Pedagogy for a Digital Age<\/i> (2nd ed.). Routledge. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780415539975.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Rethinking+Pedagogy+for+a+Digital+Age&rft.date=2013&rft.edition=2nd&rft.pub=Routledge&rft.isbn=9780415539975&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AndersonTax01-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AndersonTax01_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Anderson, L.W.; Krathwohl, D.R., ed. (2001). <i>Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives<\/i> (1st ed.). Pearson. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 978-0801319037.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Taxonomy+for+Learning%2C+Teaching%2C+and+Assessing%3A+A+Revision+of+Bloom%27s+Taxonomy+of+Educational+Objectives&rft.date=2001&rft.edition=1st&rft.pub=Pearson&rft.isbn=978-0801319037&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CurryPutting18-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CurryPutting18_33-0\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-CurryPutting18_33-1\">33.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Curry, N. (05 October 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cambridge.org\/elt\/blog\/2018\/10\/05\/putting-the-pedagogy-first-in-digital-pedagogies\/\" target=\"_blank\">\"Putting the pedagogy first in digital pedagogies\"<\/a>. <i>World of Better Learning Blog<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cambridge.org\/elt\/blog\/2018\/10\/05\/putting-the-pedagogy-first-in-digital-pedagogies\/\" target=\"_blank\">https:\/\/www.cambridge.org\/elt\/blog\/2018\/10\/05\/putting-the-pedagogy-first-in-digital-pedagogies\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 January 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Putting+the+pedagogy+first+in+digital+pedagogies&rft.atitle=World+of+Better+Learning+Blog&rft.aulast=Curry%2C+N.&rft.au=Curry%2C+N.&rft.date=05+October+2018&rft_id=https%3A%2F%2Fwww.cambridge.org%2Felt%2Fblog%2F2018%2F10%2F05%2Fputting-the-pedagogy-first-in-digital-pedagogies%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GymrekRecomm16-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GymrekRecomm16_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gymrek, M.; Farjoun, Y. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4870738\" target=\"_blank\">\"Recommendations for open data science\"<\/a>. <i>GigaScience<\/i> <b>5<\/b>: 22. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13742-016-0127-4\" target=\"_blank\">10.1186\/s13742-016-0127-4<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4870738\/\" target=\"_blank\">PMC4870738<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27195107\" target=\"_blank\">27195107<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4870738\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4870738<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Recommendations+for+open+data+science&rft.jtitle=GigaScience&rft.aulast=Gymrek%2C+M.%3B+Farjoun%2C+Y.&rft.au=Gymrek%2C+M.%3B+Farjoun%2C+Y.&rft.date=2016&rft.volume=5&rft.pages=22&rft_id=info:doi\/10.1186%2Fs13742-016-0127-4&rft_id=info:pmc\/PMC4870738&rft_id=info:pmid\/27195107&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4870738&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilsonBest14-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilsonBest14_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wilson, G.; Aruliah, D.A.; Brown, C.T. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3886731\" target=\"_blank\">\"Best practices for scientific computing\"<\/a>. <i>PLOS Biology<\/i> <b>12<\/b> (1): 1001745. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pbio.1001745\" target=\"_blank\">10.1371\/journal.pbio.1001745<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3886731\/\" target=\"_blank\">PMC3886731<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24415924\" target=\"_blank\">24415924<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3886731\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3886731<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Best+practices+for+scientific+computing&rft.jtitle=PLOS+Biology&rft.aulast=Wilson%2C+G.%3B+Aruliah%2C+D.A.%3B+Brown%2C+C.T.+et+al.&rft.au=Wilson%2C+G.%3B+Aruliah%2C+D.A.%3B+Brown%2C+C.T.+et+al.&rft.date=2014&rft.volume=12&rft.issue=1&rft.pages=1001745&rft_id=info:doi\/10.1371%2Fjournal.pbio.1001745&rft_id=info:pmc\/PMC3886731&rft_id=info:pmid\/24415924&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3886731&rfr_id=info:sid\/en.wikipedia.org:Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation attempts to remain faithful to the original, with only a few minor changes to presentation. Grammar and punctuation has been updated reasonably to improve readability. In some cases important information was missing from the references, and that information was added. The original URL for the Anaconda Distribution is dead; an archived version of the URL was used for this version. The original \"Using Interact\" URL was also broken; an updated live version was substituted for this version. The UKRN URL also changed, and a current URL is used for this version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150623\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.073 seconds\nReal time usage: 7.824 seconds\nPreprocessor visited node count: 24792\/1000000\nPreprocessor generated node count: 37243\/1000000\nPost\u2010expand include size: 159147\/2097152 bytes\nTemplate argument size: 51088\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 517.865 1 - -total\n 85.56% 443.108 1 - Template:Reflist\n 72.03% 373.016 35 - Template:Citation\/core\n 36.05% 186.702 14 - Template:Cite_journal\n 35.29% 182.737 19 - Template:Cite_web\n 7.40% 38.318 1 - Template:Infobox_journal_article\n 7.07% 36.634 1 - Template:Infobox\n 4.90% 25.366 80 - Template:Infobox\/row\n 4.74% 24.544 24 - Template:Citation\/identifier\n 4.65% 24.102 2 - Template:Cite_book\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12534-0!*!0!!en!5!* and timestamp 20210705150616 and revision id 42804\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education\">https:\/\/www.limswiki.org\/index.php\/Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","31bea4f4ab2f82a24f0d0dad004bfbb0_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/1a\/Fig1_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/84\/Fig2_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/89\/Fig3_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/90\/Fig4_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/09\/Tab1_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ea\/Fig5_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/70\/Fig6_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/da\/Fig7_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/ee\/Fig8_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/9c\/Fig9_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/64\/Fig10_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/c1\/Fig11_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/c4\/Fig12_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/83\/Fig13_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d6\/Fig14_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/ae\/Fig15_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/37\/Fig16_Davies_PLOSCompBio20_16-11.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/3b\/Fig17_Davies_PLOSCompBio20_16-11.png"],"31bea4f4ab2f82a24f0d0dad004bfbb0_timestamp":1625497575,"edf532278ba56a75b0d284a94702804c_type":"article","edf532278ba56a75b0d284a94702804c_title":"Emerging and established trends to support secure health information exchange (Spanakis et al. 2021)","edf532278ba56a75b0d284a94702804c_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange","edf532278ba56a75b0d284a94702804c_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Emerging and established trends to support secure health information exchange\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nEmerging and established trends to support secure health information exchangeJournal\n \nFrontiers in Digital HealthAuthor(s)\n \nSpanakis, Emmanouil G.; Sfakianakis, Stelios; Bonomi, Silvia; Ciccotelli, Claudio; Magalini, Sabina; Sakkalis, VangelisAuthor affiliation(s)\n \nFoundation for Research and Technology, Sapienza Universit\u00e0 di Roma, Fondazione Policlinico Universitario Agostino GemelliPrimary contact\n \nEmail: spanakis at ics dot forth dot grEditors\n \nPattichis, Constantinos S.Year published\n \n2021Volume and issue\n \n3Article #\n \n636082DOI\n \n10.3389\/fdgth.2021.636082ISSN\n \n2673-253XDistribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2021.636082\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2021.636082\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Materials and methods \n\n3.1 Current status on standards-based health data exchange \n3.2 Emerging supportive technologies: Blockchain \n\n\n4 Results and discussion \n\n4.1 A novel view on information sharing \n\n4.1.1 Federation management \n4.1.2 Data processing consent management \n4.1.3 Data sharing management \n4.1.4 Summary of recommendations \n\n\n4.2 Evaluation of emerging technical solutions: Blockchain \n\n\n5 Conclusions \n6 Acknowledgements \n\n6.1 Author contributions \n6.2 Data availability statement \n6.3 Funding \n6.4 Conflict of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nThis work aims to provide information, guidelines, established practices and standards, and an extensive evaluation on new and promising technologies for the implementation of a secure information sharing platform for health-related data. We focus strictly on the technical aspects and specifically on the sharing of health information, studying innovative techniques for secure information sharing within the healthcare domain, and we describe our solution and evaluate the use of blockchain methodologically for integrating within our implementation. To do so, we analyze health information sharing within the concept of the PANACEA project, which facilitates the design, implementation, and deployment of a relevant platform. The research presented in this paper provides evidence and argumentation toward advanced and novel implementation strategies for a state-of-the-art information sharing environment; a description of high-level requirements for the national and cross-border transfer of data between different healthcare organizations; technologies to support the secure interconnectivity and trust between information technology (IT) systems participating in a data-sharing \u201ccommunity\u201d; standards, guidelines, and interoperability specifications for implementing a common understanding and integration in the sharing of clinical information; and the use of cloud computing and prospectively more advanced technologies such as blockchain. The technologies described and the possible implementation approaches are presented in the design of an innovative secure information sharing platform in the healthcare domain.\nKeywords: interoperability, health information exchange, eHealth, blockchain, security, patient consent\n\nIntroduction \nInformation technology (IT) has long been identified as a cornerstone for efficient, cost-saving, timely, and reliable healthcare delivery.[1][2] The availability of healthcare information and patient records in digital form facilitates the persistence and posterity of valuable information and greatly supports the decision-making process, and even the extraction of new knowledge at both the individual and population levels. In our previous work, we have emphasized the current state of the art about cybersecurity in the healthcare domain, with emphasis on current threats and methodologies.[3] Paraphrasing the famous words of John Donne, \u201cno IT system is an island, entire of itself.\u201d Today, in a highly connected world where geographic boundaries have been largely eliminated and people can freely move between cities, states, countries, or continents, the requirement for two different information management systems to exchange a person's clinical data or medical history becomes vital and persistent. Sharing health information (e.g., via health information exchange [HIE]) through electronic means greatly improves the cost, quality, and patient experience of healthcare delivery.\nTo better secure the IT system's potential for interconnectivity and cooperation with other systems, the use of interoperable technologies and standards is needed. Depending on the extent and scope of the envisaged shared information spaces, there may be different levels of interoperability. Figure 1 shows a proposed \u201cmaturity\u201d model for interoperability in eHealth.[4] The model consists of five levels that, incrementally, describe a more mature version of an interoperable infrastructure, starting from Level 1 for non-connected eHealth applications; Level 2 where a single eHealth application is directly linked to another application for simple data exchange[5]; Level 3 for distributed systems that agree on protocols used, data formats, message exchange patterns, etc.[6]; Level 4, where eHealth applications from different suppliers that serve a common goal are linked but the applications do not need to have common objectives[7]; and finally, at the \u201cuniversal\u201d Level 5, where diverse eHealth applications connect to an open, interoperable infrastructure possibly spanning multiple countries.[8][9]\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 1. A maturity model for interoperability in eHealth, adapted from Van Velsen et al. 2016[4].\n\n\n\nInteroperability and data sharing in the healthcare domain is additionally challenging due to the multiplicity of the stakeholders, that is, the entities that operate (or are involved in any way) in this domain and which will be affected by any \u201cdisruption\u201d or reform of the system. Some of the most important stakeholders or actors are therefore the following:\n\n the patients who are actually treated or, in general, are the recipients of the health services;\n the medical professionals (physicians and medical personnel) to provide the medical care;\n the healthcare organizations (HCOs) (healthcare providers) as represented by their board of directors, who actually administer the health delivery from a business perspective;\n the insurance companies that provide health coverage plans;\n the pharmaceutical companies that produce and market medications to be prescribed by physicians for the treatment of patients; and\n the governments and other regulatory parties who control, coordinate, and set the rules, rights, and obligations of any involved party.\nAll these actors could have an influence in the design of a data sharing system and can also set important\u2014and conflicting in some cases\u2014requirements. For example:\n\n Patients would like to have their medical record shared, but only after their approval and only with specific authorized personnel in specific circumstances.\n An HCO can be extremely cautious about sharing the data of their patients with another organization because they are concerned by the security and availability of their systems.\n Governments of E.U. member states can impose strict laws about the transfer of their citizens in cross-border healthcare treatment scenarios.\n Medical professionals require fast and effortless access to a patient's medical history in emergency situations, which cannot be the case if time-consuming authorization processes are the norm. \nGiven these and similar requirements, and even though the objective is to design a technical solution for the sharing of clinical data, it is imperative that all these constraints and requirements are considered and addressed in a satisfying manner.\nFrom a strictly technical point of view, the sharing platform may need to be interoperable with a large number and diverse set of IT systems, each with their own protocols, data formats, etc. Some of the most important systems that manage patient-related data, and could be used as data sources for information sharing, include electronic health records (EHRs), personal health records (PHRs), laboratory information systems (LISs), and picture archiving and communication systems (PACS). EHRs are patient-centered systems that store and manage clinical information, such as a patient's medical history, diagnoses, medications, immunization dates, allergies, radiology images, and lab and test results. They are managed by authorized personnel, usually in the context of a single HCO, although they can span beyond that. PHRs are electronic applications that are used by people managing their own health information in a private and confidential environment. They are simpler systems than EHRs, and in some cases, they can be connected (temporarily or otherwise) to more enterprise-level HCO systems (e.g., EHRs or other hospital information systems). An LIS is used inside hospitals and clinics to record, manage, and store data for clinical laboratories in a patient-centric way (e.g., sending laboratory test orders to lab instruments, tracking those orders, and then recording the results in a searchable database). And PACS are systems used in a clinical setting for the storage and convenient access to medical images from multiple modalities (source machine types). Digital Imaging and Communications in Medicine (DICOM) is the standard format and suite of protocols for the storage and transfer of images from PACS services. Of course, in a health ecosystem there may be additional systems, for example, for the management of insurance claims and for clinidal diagnoses (e.g., clinical decision support systems [CDSSs]).\nIn the general context, information sharing involves more than one party (healthcare providers, organizations, etc.,) that need to cooperate and agree on the way the exchange of information happens, and what the rules and policies are that govern it. Interoperability involves many different aspects, such as legislation and guidelines, contracts, and agreements between exchanging parties, governance and maintenance, shareable workflows, standardized data elements, semantic and syntactic choices, applications, technical infrastructure, and safety and privacy issues. The Refined eHealth European Interoperability Framework (EIF) is a set of recommendations that specify the standards, protocols, procedures, and policies that when deployed can improve the interoperability of eHealth applications within the E.U. and across its member states by providing specific recommendations for all these aspects.[10] Figure 2 depicts how these aspects can be represented in interoperability \u201clevels\u201d that permit two different organizations to communicate.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 2. Alignment of interoperability levels between two communicating organizations.\n\n\n\nThis framework provides a great overview of the needed \u201cglue\u201d so that two or more healthcare environments can collaborate and serve as a common, multilevel, and multi-perspective model on the interoperability requirements. The six different levels of the (refined) EIF are:\n\n Legal and regulatory: This level represents the legislation and regulatory guidelines that define the boundaries for interoperability not only across borders, but also within a country or region.\n Policy: This level represents the contracts and agreements between the sharing organizations so that trust is established and responsibilities are assigned.\n Care process: This level represents the shared workflows that define how the integrated care is delivered and how these workflows are managed.\n Information: This level defines the data models, the concepts and their values, the terminologies, and controlled vocabularies that cater for the common understanding of the exchanged electronic messages.\n Applications: This level defines how the data are extracted from and imported to the healthcare information systems and how the transport of the data takes place using health-specific technologies and standards.\n IT infrastructure: This level represents the lowest level and corresponds to general-purpose communication and network protocols.\nThis generic EIF is highly relevant for the use cases of the health information sharing since sharing is greatly facilitated between interoperable systems and organizations. Here, we focus on the sharing of health-related information across HCOs and even across countries and continents. This is an important use case to improve the secure and efficient delivery of health care across Europe.[11][12] The importance of cross-border health care in Europe has been recognized since 2011's Directive 2011\/24\/EU, which established patients' rights to access safe and high-quality healthcare, including across national borders within the E.U., and their right to be reimbursed for such healthcare.[13] As can be deduced by considering the Refined eHealth EIF (Figure 2), the cross-border sharing of clinical information is a complex scenario due to the fact that data need to be transferred between different countries and therefore, requires overcoming barriers such as establishing a common trust framework, uniquely identifying citizens, and translating between different schemas and terminologies. This paper presents current approaches to address most of these issues in the European context while presenting and evaluating emerging technologies such as blockchain that have been in the limelight recently. Our objective is to take advantage of both well-established and novel technologies that complement each other in order to design an architecture for the secure exchange of clinical information in the European context.\n\nMaterials and methods \nCurrent status on standards-based health data exchange \nIn the healthcare industry, large standards developing organizations have defined numerous standards, data formats, terminologies, and more in order to support the design and building of interoperable IT systems. Perhaps the most well-known and most important standards are the ones introduced by Health Level 7 (HL7) and SNOMED CT, which can be used as a foundation for the development of data exchange standards among eHealth systems.[14] Two more standards organizations are Integrating the Health Enterprise (IHE), which focuses primarily on integration and interoperability, and the Clinical Data Interchange Standards Consortium (CDISC). CDISC produced the Operational Data Model (ODM) to \u201cfacilitate the regulatory-compliant acquisition, archive, and interchange of metadata and data for clinical research studies.\u201d[15] ODM is an XML-based format that provides a number of constructs for modeling electronic case report forms (CRFs) and can also be used in sending forms data from a clinical trial system to an EHR system. In the area of medical devices, the Continua Health Alliance, a non-profit, open-industry coalition of healthcare and technology companies working to establish a system of interoperable personal health solutions, develops an ecosystem of connected technologies, devices, and services that will enable the more efficient exchange of fitness, health, and wellness information.[16] Among its proposed standards, Continua proposes specifications and standards such as Bluetooth, USB, ISO\/IEEE 11073 (for medical devices), and HL7 to enable patients to use home-based devices to monitor their weight, blood pressure, and glucose and blood oxygen levels, as well as to share these data with their healthcare professionals.\nThe exchanged information can be in multiple data formats based on the type of data, device category, etc.[17] Some of the most common formats based on the health applications using them are the following:\n\n For medical imaging, the use of DICOM is almost universal and defines not only the content (DICOM file format) but also communication protocols for the exchange of medical images.[18] \n In the area of DNA sequencing and other -omics data formats, we have FASTQ file format[19], which is used to store sequence information, and the standard flowgram format (SFF), which is used to encode sequence reads.[20]\n The majority of the EHR systems adopt the HL7 standard clinical document architecture (CDA) as the interoperable data format.[21] CDA is part of the HL7 Version 3 family, and it is based on a reference information model (RIM) that serves as a semantic model that consists of a set of structural components (e.g., classes with data types) and semantic relations that are used to represent clinical notes in the form of an extensible markup language document.\nThe use of controlled vocabularies and terminologies allows for the unambiguous representation of important value sets such as \"diagnosis\" and \"prescribed medicines.\"[22] The following are examples of such terminologies:\n\n The International Classification of Diseases (ICD) provides a common language for reporting and monitoring diseases, used throughout the world to compare and share data in a consistent standard way between hospitals, regions, and countries and over periods of time. It is used to classify diseases and other problems for payment, management, and research, as recorded on many types of health records including medical records and death certificates. ICD-11 is the latest version of it, whereas ICD-10 (released in 1993) remains widely used.\n SNOMED CT, already mentioned above, is the most comprehensive multilingual clinical healthcare terminology available. It is used in EHR systems to facilitate clinical documentation and reporting and to retrieve and analyze clinical data. SNOMED CT is both a coding scheme, identifying concepts and terms, and a multidimensional classification, enabling concepts to be related to each other, grouped, and analyzed according to different criteria.\n Logical Observation Identifiers Names and Codes (LOINC) provides a set of universal identifiers for medical laboratory observations. LOINC provides codes for the observation names (e.g., eye color), not the observation finding (e.g., blue eyes). LOINC therefore provides codes for questions, and where needed, other vocabularies, such as SNOMED CT, provide codes for the answers.\n The Unified Medical Language System (UMLS) is an important terminology resource, intended for use mainly by developers of health information systems. The UMLS \u201cMetathesaurus\u201d uses several different source vocabularies and seeks to reflect and preserve the meanings of concept names and relationships from these sources. It is therefore a valuable resource for the translation between the different source vocabularies.\nApplication-level interfaces are also needed to support the communication and exchange of the standards-based encoded information. The role of HL7 is principal on this front: HL7's name comes from \u201cLevel Seven,\u201d which, according to the Open Systems Interconnection (OSI) model that standardizes communication functionality in IT, corresponds to application layer. From its establishment in the late 1980s, HL7 was therefore focused on exchanging information within hospitals. The focus remains almost the same today, but HL7 has progressed from different paradigms over the years, in order to describe the structure, semantics, and management of the exchanged information. The development of HL7 Version 3 (HL7v3) started around 1995 in order to introduce more consistency between the implementations of Version 2 following an object-oriented development methodology. The most recent proposal by HL7 is Fast Healthcare Interoperability Resources (FHIR), which leverages web technologies to overcome the complexity of HL7v3.[23]\nOn the other hand, the IHE initiative has defined a number of \u201cintegration profiles,\u201d which are detailed specifications for communication among systems to address key clinical use cases, all based on established standards. IHE profiles organize and leverage the integration capabilities that can be achieved by coordinated implementation of communication standards, such as DICOM, HL7, W3C, and other security standards.[24] Some of the IHE integration profiles that might be interesting in the context of interfacing health information systems and sharing of clinical information include:\n\n Cross-Enterprise Document Sharing (XDS): Allows EHR documents to be shared and discovered among healthcare enterprises, physician offices, clinics, acute care in-patient facilities, and PHRs.\n Patient Demographics Query (PDQ): Enables applications to query by patient demographics (e.g., name) for patient identity from a central patient information server.\n Patient Identifier Cross Referencing (PIX): Allows applications to query for patient identity cross-references between hospitals, sites, HIE networks, etc.\n PDQ HL7 v3 (PDQv3): Extends the PDQ profile leveraging HL7 Version 3.\n PIX: Extends the Patient Identifier Cross-Reference profile leveraging HL7 Version 3.\n Cross-Community Access (XCA): Allows the query and retrieval of patients' EHRs held by other communities.\n Cross-Enterprise Document Reliable Interchange (XDR): Exchanges health documents between health enterprises using a web service-based point-to-point push network communication.\nThere are two main architectural approaches for the implementation of an information sharing platform: a centralized approach and a federated approach.[25] In the centralized approach, a central data warehouse and accompanying services act as middlemen for the exchange of information and a single source of patient data that are shared among the participating organizations. On the other hand, in the federated architecture, a central infrastructure is also in place, but in this case, it merely acts as a facilitator for locating the data sources. An example of this case would be a common registry that stores only the links to the original patient records, medical images, and other types of data while the linked data are not transferred outside their primary premises unless explicitly requested by any interested client system. In addition to these opposite approaches for designing a distributed information sharing platform, there are also various hybrid options, such as using messaging with \u201cpublish-subscribe\u201d communication that can be introduced to complement either the centralized or federated architectures.\nThere are advantages and disadvantages in all of the abovementioned deployment options. For example, in the federated approach, there are more strong concerns about the privacy, security, and availability of the data shared and their original sources.[26] The operation of a mission-critical radiology information system (RIS) in a hospital can be severely affected if multiple peers request DICOM images from its PACS, and this poses an additional burden and cost for the acquisition and management of adequate infrastructure in the source organization. Instead, a centralized strategy allows for easy access to the whole information shared but also leads to a concentration of the costs for maintaining the infrastructure needed and can be problematic at the operation level (a \u201csingle point of failure\u201d). Furthermore, there are more costs on integrating the different data sets under a common \u201cschema,\u201d resolving conflicts or even supporting the timely update of the persisted information when a source system acquires new or modified data.\n\nEmerging supportive technologies: Blockchain \nBlockchain is a decentralized, distributed data structure used to store transactions (aggregated in blocks) across many computers.[27][28] Blockchain has been extensively used for Bitcoin.[29] In the realm of healthcare, we have seen Kuo et al.[30] perform a systematic review on how blockchain can be used in healthcare applications. Blockchain core is the embedded distributed ledger technology able to support data integrity, authenticity, and origin. In blockchain, each block is linked to the previous one through a cryptographic hash, and each block is a data structure that allows the storage of a list of transactions.[31] In the blockchain, a transaction abstracts and allows the tracking of an exchange or interaction between two entities. Transactions are created and exchanged by peers of the blockchain network and modify the state of the blockchain data structure. An efficient categorization and comprehensive overview of the latest privacy-preserving mechanisms and policies regarding smart electric grids, focusing on the use of the blockchain technology and the multi-authority access control paradigm, is offered by Triantafyllou et al.[32] and Radoglou et al.[33]\nConcerning data access, we can have the following:\n\n Public blockchain: There are no restrictions on reading blockchain data and submitting transactions for inclusion into the blockchain.\n Private blockchain: Direct access to blockchain data and submitting transactions is limited to a predefined list of entities.\nConcerning data management, we can have the following:\n\n Permissioned blockchain: Transaction processing is performed by a predefined list of peers with known identities.\n Permissionless blockchain: No restrictions on identities of transaction processors (i.e., blocks creators).\nCombining the two perspectives, we can have four categories, as depicted in Figure 3.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 3. Blockchain classification overview.\n\n\n\n\r\n\nBlockchain supports auditability and transparency, as any reader is able to verify the correctness of the state of the system. Indeed, by storing all the transactions, it is possible to replay (starting from a correct checkpoint) the entire history and check that the current state is consistent with the set of recorded transactions. It is important to note that, when using a blockchain to store data, there exists an inherent tradeoff between transparency and privacy. Indeed, if the primary requirement is to have a fully transparent system, we need to accept that anyone is allowed to see any piece of information (sacrificing privacy). Conversely, if the primary requirement is to have a private system, it will not provide transparency. A tradeoff between transparency and privacy is however possible, but it will come at the cost of efficiency, as it would require employing complex cryptographic primitives.\n\nResults and discussion \nA novel view on information sharing \nNowadays, the sharing of a patient's clinical information between two HCOs (e.g., hospitals) usually requires a great amount of manual work in order to check and validate patient's consent (by consulting signed papers) or, at worst, results in privacy loss by extending the trust circle, e.g., to all physicians from the requesting organization. The main limitation of the current sharing pattern can be summarized in the following way:\n\n Sharing medical data may require time and possibly multiple interactions, also involving the patient in the loop.\n Sharing medical data is currently a physical point-to-point interaction. If the same set of data needs to be shared with multiple parties, it would require multiple sharing patterns to be in place.\n Sharing is currently asymmetric. Organization A may have the consent to share patient's data with organization B, but the inverse may not be true.\nIn order to overcome these deficiencies, we aim to design a platform\u2014the Innovative Secure Information Sharing Platform (InSISP)\u2014that is able to support a fast and efficient medical information sharing at both national and cross-national levels, taking into account sharing constraints, including those imposed by the General Data Protection Regulation (GDPR). It is imperative that patient's consent is central in this framework, and one of the challenges is to make the consent management robust, simple, and secure. A sequence diagram of the possible interactions to support the data sharing is shown in Figure 4.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 4. InSISP sharing pattern for medical data.\n\n\n\nFrom an abstract point of view, InSISP can be seen as a data repository that can be accessed by HCOs (i.e., clients) to store and retrieve shared data by using a common interface and format (e.g., CDA Release 2 using standard vocabularies such as SNOMED and LOINC). The shared data repository is surrounded by a federation of collaborating entities (organizations\/clients) that is dynamic and evolving. Once the federation is established, participating entities can start sharing data according to the data processing consent provided by patients. To this aim, we can identify two additional functionalities: (i) data sharing and (ii) data processing consent management. Figure 5 summarizes a possible decomposition of the InSISP and highlights the three storage components.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 5. InSISP functional decomposition.\n\n\n\nFederation management \nThe federation management functionality has the aim to manage the federation life cycle, and in particular, it should allow new HCOs to join and HCOs no longer interested in participating in the federation to leave. In particular, this functionality supports the following operations:\n\n Join the federation: HCOs should be able to become part of the federation at any time. When considering a membership service, all the members are assumed to be uniquely identified. So from the perspective of the InSISP development, there should be an external service that is able to support the identification of members and to provide them with a digital identity.\n Leave the federation: HCOs may decide to leave the federation at their will, and the InSISP should support the removal of the entity from the membership and should notify the end of the sharing to connected entities.\n Get the federation membership: Allows an HCO participating in the federation to get the current membership and know the set of HCOs potentially involved in the sharing, including their identification, public keys, and the categories of data that are currently available for the sharing.\nFederation management intrinsically relies on the execution of a distributed protocol running, and thus, there are two main options to implement a federation membership service: (i) client\/server or (ii) peer to peer. In the client\/server case, the current membership of the system is maintained by a trusted third party (TTP). When a new HCO wants to join the federation, it simply needs to contact the TTP and identify and authenticate itself with the TTP that will proceed by adding it to the current view. Similarly, when an HCO in the federation wants to leave, it simply needs to notify the TTP that will remove it from the current view. The current membership can be obtained again by querying the TTP. The main advantage of this option is that all the complexities of the membership management are delegated to the TTP. However, this also implies that the TTP is clearly a single point of failure for the system as well as its main bottleneck. In the peer-to-peer approach, HCOs collaborate to maintain a consistent view of the system by exchanging messages and trying to reach a consensus on the sequence of views generated to include new members and to remove old ones.\nChockler et al.[34] discussed in detail the formalization and the specification of the group membership service, while more recently, Aguilera et al.[35] considered the problem of building a reconfiguration service to support the development of a distributed shared storage. However, let us note that in all these cases, the emphasis is on how to provide a consistent view to all the members. To the best of our knowledge, there does not exist any approach investigating the cost of realizing a membership service using blockchain technologies.\nThe main advantage of the peer-to-peer approach is its intrinsic resilience. In addition, in peer-to-peer settings, it is also possible to consider a blockchain-based approach to construct the sequence of consistent views providing the \"view auditability\" property for free. The main drawback is the cost imposed by the management of consistent views, as it requires to run coordination and synchronization protocols among all the participants that would bring poor scalability in case of a highly dynamic federation.\n\nData processing consent management \nThe data processing consent management function has the aim to support the development of a digital data processing activity registry to store and access patients' data processing consent. The data processing consent is granted by a patient for a specific set of data to a specific set of entities and for a specific purpose and period of time. In order to support the automatic verification of patients' consent, such information must be stored and managed by the HCO in an electronic form (e-Consent).[36][37][38] Also, every patient has the right to modify their consent and has the right to be forgotten; i.e., at any point in time, they may ask to revoke all their previous consent (while also erasing any data identifying them).\nTo this aim, the data processing consent management functionality should offer the following operations:\n\n Provide new consent: It should allow the addition of a new entry to the table, specification of the beneficiary entities, and seting of the expiration time of the consent.\n Update consent: It should allow the modification of existing consent by allowing sharing with a new beneficiary or by removing a beneficiary.\n Remove (all) consent: It should basically implement the right to be forgotten by deleting all the consent previously provided by a specific patient.\nDiscussion about possible design and deployment options\nLet us note that the data processing consent management function supports every HCO in managing its own data processing activity registry. Thus, from this point of view, we can say that it is local to every HCO.\nAs a consequence, the most appropriate choice is to design it as a local data store managed and accessed only by one HCO. Of course, in order to increase the resiliency and security of the storage, it can be also replicated, but all the replicas will still be managed by the same HCO.\nHowever, a distributed design raises a privacy issue in patients' information. Indeed, even if data in the registry are not sensible by themselves, they could be easily correlated to infer sensitive information about patients and would result in a privacy violation; e.g., by looking to the list of HCOs where Bob did his analysis, you may infer that Bob is affected by a specific disease. To solve this issue, it is necessary to employ anonymization scheme generation, an extra cost without any particular advantage in terms of reliability or security.\n\nData sharing management \nData sharing is the core functionality of the InSISP, as it manages the real transfer of medical data between parties. It offers just one main operation, i.e., Get Data, which is used to retrieve a specific piece of data for a specific patient and transfer it according with the patient's consent.\nWe can consider three main options to design and deploy this functionality: client\/server, peer-to-peer with message exchanges, and peer-to-peer with shared memories. In the client\/server case, data available for sharing are copied and pushed toward a centralized TTP that will take care of satisfying the sharing request. In order to be GDPR-compliant, a specific consent to move data to the TTP must be signed, as well as the consent to share data with all the federation members. (Let us note that this second set of consent could be removed if every HCO provides the TTP a copy of its data processing activity registry. However, as mentioned above, this would add the complexity of finding a good anonymization scheme that allows to preserve patients' privacy still allowing the TTP to check consent.) Figure 6 shows an overview of a possible client\/server design.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 6. Overview of the client\/server design with a trusted third party (TTP).\n\n\n\nIn the peer-to-peer case, the idea is that the sharing is realized by letting HCOs in the federation cooperate with each other. We can distinguish two cases: cooperation realized through message exchange and cooperation realized through shared memory.\nIn the message exchange case, the sharing is realized using an ad hoc request\u2013reply communication pattern, as shown in Figure 7.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 7. Overview of the peer to peer with message exchange design.\n\n\n\nIn the peer to peer with shared memory design (Figure 8), each HCO creates a shared memory space where it stores all the pieces of data that can be shared according to the data processing activity registry. As an example, let us consider the case where Bob provided the consent to HCO1 to share his X-ray images with HCO2. This means that HCO1 and HCO2 create locally a shared space where they will store a copy of all Bob's X-ray images (the red slice in Figure 8).\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 8. Overview of the peer to peer with shared memory design.\n\n\n\nDiscussion about possible design and deployment options\nTechnically, all the three considered designs are feasible. However, as anticipated in the previous section, the client\/server scenario poses several challenges from the point of view of the consent needed in order to make it compliant. In particular, the main issue is the large set of consent data that is necessary, and that patients may be reluctant to provide consent in the first place. Instead, the two point-to-point designs solve this issue, as they exploit locally the consent information and move data only toward authorized HCOs.\n\nSummary of recommendations \nAccording to the considerations discussed in the previous sections, Table 1 summarizes the viable options for the design of each functionality of the InSISP.\n\r\n\n\n\n\n\n\n\n\n\n\nTable 1. Summary of design options and recommendations.\n\n\n\nEvaluation of emerging technical solutions: Blockchain \nIn order to evaluate if blockchain is a valid option to support the InSISP deployment, the methodology presented by Staderinin et al.[39] has us considering the following three steps:\n\n Perform a requirements analysis to assess blockchain benefits.\n Evaluate the most appropriate blockchain solution where the designer is guided on the choice of the most suitable blockchain category, based on blockchain-specific criteria, depending on who are readers and writers of the data and who is allowed to generate data.\n Select the blockchain configuration, which assists the designer throughout the decision-making process for the configuration of the blockchain compliantly with the chosen category and the given project requirements.\nLet us remark that blockchain is intrinsically a distributed system, and it makes no sense trying to use it when a distributed setting is not appropriate, which means that it can be used for the federation membership and for the data sharing functionalities of InSISP. In the following, we will evaluate the suitability of blockchain-based solution by adoption of the methodology described by Aguilera et al.[35]\nIn fact, trying to evaluate the blockchain technologies for the data sharing scenarios and design solutions described above, the first step is the analysis of requirements related to the component under analysis in order to understand the benefit of adopting a blockchain-based solution for its low-level design and development. The factors that are considered in this step include:\n\n Data or state storage: The first element to consider is to check if the module under analysis needs to store data or system states. If no information needs to be stored, clearly no blockchain is needed.\n Immutability and data integrity: With immutability, we refer to the property of a piece of data to never change (i.e., a constant value that is never updated). If immutability is a requirement, then blockchain is certainly an option, as this is probably the most distinctive property of any blockchain. Integrity is strictly related to immutability, and this is why they are analyzed together, also considering that they are both closely related to cryptography. If a component requires data protection from unauthorized modifications, then this requirement can be met with blockchain.\n Non-repudiation: Non-repudiation means that the author of some message\/data cannot deny that it produced the message. This is another fundamental property that can be easily satisfied using blockchains.\n Multiple writers. This criterion considers the multiplicity of entities in charge of writing data in the storage. If only one entity is a writer, then a common database is probably more appropriate than a blockchain, especially from the performance perspective, i.e., in terms of throughput and latency.\n TTP always online: A TTP is an entity that facilitates interactions between mutually mistrusting entities. If in the system a TTP is required and it is planned to be always online, entities can delegate write operations to it as transactions, or as state changes. Therefore, the TTP plays the role of a trusted deliverer and verifier. In this case, a blockchain, known for being a trustless technology, becomes useless, and the methodology brings to the related output. Otherwise, it can happen that the involvement of a TTP is planned but not for being always online: in this case, it could play the role of an authority giving authorizations for permissioned blockchains. Alternatively, a TTP may not exist at all. In the latter situations, it is not possible to exclude the recommendation of using a blockchain.\n Writers are known and trusted If all the entities interested in writing know and mutually trust each other, a blockchain is superfluous and not recommended (again mainly for performance issues).\nThe flowchart of this analysis step is shown in Figure 9. Tables 2 and 3 then answer the relevant questions for the assessment of blockchain in the federation management and data sharing operations according to the decision flowchart in Figure 9. \n\r\n\n\n\n\n\n\n\n\n\n\nFigure 9. Flowchart: blockchain, yes or no? adapted from Staderini et al. 2018[39].\n\n\n\n\n\n\n\n\n\n\n\n\nTable 2. Assessment of blockchain in the federation management.[39].\n\n\n\n\n\n\n\n\n\n\n\n\nTable 3. Assessment of blockchain for data sharing operations.\n\n\n\nGoing back to the maturity model flowchart in Figure 1, we at first glance may be led to believe that blockchain is not recommended to support the implementation of the federation membership mainly because there exists a basic level of trust between members of the federation. In addition, we may be led to assume that the federation membership is also relying on a trusted external service providing digital and secure identities to participants. However, we must highlight the opportunity to consider a non-repudiation requirement; as such, we considered the importance of preserving data integrity (i.e., to ensure that the current membership cannot be altered). Thus, if these two requirements become more relevant or if assumptions on the identity platform or about trustworthiness of participant cannot be met, then blockchain immediately becomes a viable solution.\nUsing the same process, at first, we are led to believe that blockchain is recommended to support the implementation of data sharing mainly because it would efficiently support the data integrity requirement, i.e., it allows the traceability of data accesses and verifies their authorship and integrity. However, we should keep in mind that medical data are mostly read-only, and in our context, they are shared between trusted parties. Thus, the main benefit we can get by adopting a blockchain-based solution is the support for data integrity verification and data auditability. Yet even still, this feature must be carefully balanced with the \u201cright to be forgotten\u201d requirement (i.e., a MUST requirement imposed by the GDPR regulation) and its implication on the adoption of a blockchain-based solution. In order to support the implementation of the \u201cright to be forgotten,\u201d we need to guarantee that data can be deleted from the blockchain when the data owner asks for it to be done. Currently, deleting data efficiently from a blockchain is still an open research problem, and the few existing solutions are currently based on the adoption of computationally expensive cryptographic techniques. Furthermore, when dealing with medical data, there is also the additional complexity following the huge heterogeneity of data to be considered (i.e., text, images, and images\/sounds). Blockchain technologies have been originally designed to deal with transactional data, of small size and in the form of numbers or strings. Currently, it is not clear how to extend the paradigm to work with heterogeneous data. A possible solution to this issue could be to keep such heterogeneous medical data stored locally in a classical database and store in the blockchain only its hash. However, it is still not clear how much privacy lawyers consider such metadata as an expression of a personal data, and thus, the issue may remain.\n\nConclusions \nInformation sharing in the health domain is a complex and challenging process given the many stakeholders involved, the different and sometimes competing standards and solutions to choose from, and the important security, ethics, and regulation-related constraints for any proposed solution to comply with any \"security as a service\" attempts, e.g., as outlined by Markakis et al.[40] HIE is a key building block for the realization of Connected Health in Europe, which \u201cspeaks to the health journey of the person, through the entire lifespan, leveraging a variety of technologies to do so.\u201d[41] Based on what we've discussed in this work, it is important to consider the following aspects when building a new platform for sharing medical information:\n\n Patient consent is of utmost importance, and infrastructure should be in place for its registration, enforcement, and withdrawal.\n Compliance with GDPR and national laws is vital, as is implementing solutions to address significant requirements, such as the \u201cright to erasure.\u201d\n Effective interoperable solutions are produced by linking and integrating with well-established standards such as document and data formats (e.g., CDA and DICOM) and metadata and value lists (e.g., SNOMED and LOINC) in order to support common understanding and integration.\n Such solutions should be able to andle the whole security spectrum, including authentication and authorization of users, as well as support for data privacy, audits for \u201cpost-mortem\u201d analysis and non-repudiation, data integrity, and machine-enforced trust among the sharing organizations.\n Such solutions will also enable the unique identification of patients while at the same time exposing the minimal set of personal information in order to protect their privacy.\n Performance, scalability, and availability of the whole platform should be high in order to support the health-related processes efficiently.\n The solution should be part of the healthcare ecosystem, which means allowing easy integration with existing infrastructure by featuring interoperable \u201cports and adapters\u201d interfaces.\nAdditionally, the IHE profiles should not be neglected. In the 2015\/1302 Commission Decision, after consulting the European multistakeholder platform on information and communications technology (ICT) standardization and sectoral experts, 27 IHE profiles have been identified for referencing in public procurement, such as XDS, XDS-I, PIX, and PDQ.[42]\nCloud computing is now used everywhere and provides an important set of features, such as, adaptive scalability, performance, and benefits from the business perspective. In particular, the sharing of clinical information in the cloud can be very advantageous, especially in cases where central repositories or central coordination are needed. But organizations should also be wary about the data protection, privacy, and access control mechanisms that should be in place[43][44][45], either offered by the cloud provider or built in-house, in order to properly handle sensitive data and comply with regulations such as GDPR.\nBlockchain is a highly interesting technology that can be put to good use in information sharing, more specifically for supporting decentralization, data integrity verification, and data auditability. These inherent features of blockchain have been praised and discussed in the context of healthcare as valuable tools.[41][46] Nevertheless, there are some major issues to be resolved, such as the compliance with GDPR's \u201cright to be forgotten\u201d requirement, which, unless the blockchain implementation is adapted, requires the deletion of data from the blockchain when the data owner asks to do it, which is not feasible, by design, in the \u201ctraditional\u201d blockchain implementations. Furthermore, there is also the additional complexity followed by the huge heterogeneity of medical data (i.e., text and images) that do not fit exactly to the original design of blockchain. It is evident that such requirements imposed by GDPR and the application domain present challenges and introduce additional trade-offs related to the management of data, administration, and overall governance. For example, storing the health data \u201coff-chain\u201d (i.e., external to the blockchain network) and only metadata \u201con-chain\u201d may introduce problems of availability, performance, data protection, and integrity.[47] Therefore, careful considerations of the available options should be made before committing to such cutting-edge technologies.\n\nAcknowledgements \nAuthor contributions \nES, SS, SB, and CC made core contribution to the design of the study, collection of information, guidelines, established practices, and standards and made an extensive evaluation on new and promising technologies for the implementation of a secure information sharing platform for health-related data including blockchain evaluation. SM and VS coordinated the work in respect of the project PANACEA. All authors contributed to the article and approved the submitted version.\n\nData availability statement \nThe original contributions generated for the study are included in the article\/supplementary material; further inquiries can be directed to the corresponding author\/s.\n\nFunding \nThis work has been supported by the PANACEA project, which has received funding from the European Union's Horizon 2020 research and innovation program under the Grant Agreement No. 826293.\n\nConflict of interest \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 U.S. Congress, Office of Technology Assessment (September 1995). \"Bringing Health Care Online: The Role of Information Technologies\" (PDF). U.S. Government Printing Office. https:\/\/ota.fas.org\/reports\/9507.pdf .   \n\n\u2191 Kolodner, R.M.; Cohn, S.P.; Freidman, C.P. (2008). \"Health Information Technology: Strategic Initiatives, Real Progress\". Health Affairs 27 (Supp. 1): w391\u20135. doi:10.1377\/hlthaff.27.5.w391.   \n\n\u2191 Spanakis, E.G.; Bonomi, S.; Sfakianakis, S. (2020). \"Cyber-attacks and threats for healthcare - A multi-layer thread analysis\". Proceedings of the 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society: 5705\u201308. doi:10.1109\/EMBC44109.2020.9176698. PMID 33019270.   \n\n\u2191 4.0 4.1 Van Velsen, L.; Hermens, H.; d'Hollosy, O.-N. (2016). \"A maturity model for interoperability in eHealth\". Proceedings of the IEEE 18th International Conference on e-Health Networking, Applications and Services: 1\u20136. doi:10.1109\/HealthCom.2016.7749533.   \n\n\u2191 Chronaki, C.E.; Chiarugi, F.; Mavrogiannaki, E. et al. (2003). \"An eHealth platform for instant interaction among health professionals\". Proceedings of the Computers in Cardiology 2003: 101\u20134. doi:10.1109\/CIC.2003.1291100.   \n\n\u2191 Spanakis, M; Lelis, P.; Chiarugi, F. et al. (2005). \"R&D Challenges in Developing an Ambient Intelligence eHealth Platform\". IFMBE Proceedings 11 (1): 1727\u2013983. http:\/\/139.91.210.27\/CBML\/PROCEEDINGS\/2005_EMBEC\/Embec%202005\/Abstracts\/Abstract791.html .   \n\n\u2191 Tsiknakis, M.; Spanakis, M. (2010). \"Adoption of innovative eHealth services in prehospital emergency management: A case study\". Proceedings of the 10th IEEE International Conference on Information Technology and Applications in Biomedicine: 1\u20135. doi:10.1109\/ITAB.2010.5687752.   \n\n\u2191 Spanakis, E.G.; Psaraki, M.; Sakkalis, V. (2018). \"Congestive Heart Failure Risk Assessment Monitoring through Internet of things and mobile Personal Health Systems\". Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society: 2925-28. doi:10.1109\/EMBC.2018.8513024. PMID 30441013.   \n\n\u2191 Spanakis, M.; Sfakianakas, S.; Sakkalis, V. et al. (2019). \"PharmActa: Empowering Patients to Avoid Clinical Significant Drug\u207bHerb Interactions\". Medicines 6 (1): 26. doi:10.3390\/medicines6010026. PMC PMC6473432. PMID 30781500. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6473432 .   \n\n\u2191 \"eHealth Network: Refined eHealth European Interoperability Framework\" (PDF). European Union. 23 November 2015. pp. 24. https:\/\/ec.europa.eu\/health\/sites\/default\/files\/ehealth\/docs\/ev_20151123_co03_en.pdf .   \n\n\u2191 Katehakis, D.G.; Pangalos, G.; Prentza, A. (2017). \"Security Improvements for Better and Safer Cross-Border ePrescription and Patient Summary Services\". International Journal of Reliable and Quality E-Healthcare 6 (1): 2. doi:10.4018\/IJRQEH.2017010102.   \n\n\u2191 Tinholt, D.; Carrara, W.; Tol, T. et al. (18 June 2013). \"Final Report - Study on Analysis of the Needs for Cross-Border Services and Assessment of the Organisational, Legal, Technical and Semantic Barriers (SMART 2011\/0074)\". European Commission. doi:10.2759\/10003. https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/final-report-study-analysis-needs-cross-border-services-and-assessment-organisational-legal .   \n\n\u2191 Peeters, M. (2012). \"Free movement of patients: Directive 2011\/24 on the application of patients' rights in cross-border healthcare\". European Journal of Health Law 19 (1): 29\u201360. doi:10.1163\/157180912x615158. PMID 22428388.   \n\n\u2191 Benson, T. (2012). Principles of Health Interoperability HL7 and SNOMED (2nd ed.). Springer. doi:10.1007\/978-1-4471-2801-4. ISBN 9781447128014.   \n\n\u2191 Kuchinke, W.; Alerts, J.; Semler, S.C. et al. (2009). \"CDISC standard-based electronic archiving of clinical trials\". Methods of Information in Medicine 48 (5): 408\u201313. doi:10.3414\/ME9236. PMID 19621114.   \n\n\u2191 Carroll, R.; Cnossen, R.; Schnell, M. et al. (2007). \"Continua: An Interoperable Personal Healthcare Ecosystem\". IEEE Pervasive Computing 6 (4): 90\u201394. doi:10.1109\/MPRV.2007.72.   \n\n\u2191 Marias, K.; Sakkalis, A.; Roniotis, C. et al. (2009). \"Clinically Oriented Translational Cancer Multilevel Modeling: The ContraCancrum Project\". Proceedings of the 2009 World Congress on Medical Physics and Biomedical Engineering: 2121\u201327. doi:10.1007\/978-3-642-03882-2_564.   \n\n\u2191 Larobina, M.; Murino, L. (2014). \"Medical Image File Formats\". Journal of Digital Imaging 27: 200\u201306. doi:10.1007\/s10278-013-9657-9.   \n\n\u2191 Cock, P.J.A.; Fields, C.J.; Goto, N. et al. (2010). \"The Sanger FASTQ file format for sequences with quality scores, and the Solexa\/Illumina FASTQ variants\". Nucleic Acids Research 38 (6): 1767\u201371. doi:10.1093\/nar\/gkp1137. PMC PMC2847217. PMID 20015970. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2847217 .   \n\n\u2191 Leinonen, R.; Sugawara, H.; Shumway, M. et al. (2011). \"The sequence read archive\". Nucleic Acids Research 39 (DB1): D19\u201321. doi:10.1093\/nar\/gkq1019. PMC PMC3013647. PMID 21062823. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3013647 .   \n\n\u2191 Dolin, R.H.; Alschuler, L.; Beebe, C. et al. (2001). \"The HL7 Clinical Document Architecture\". JAMIA 8 (6): 552\u201369. doi:10.1136\/jamia.2001.0080552. PMC PMC130066. PMID 11687563. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC130066 .   \n\n\u2191 Freitas, F.; Schulz, S.; Moraes, E. (2009). \"Survey of current terminologies and ontologies in biology and medicine\". RECIIS 3 (1): 239\u201349. doi:10.3395\/reciis.v3i1.239en.   \n\n\u2191 Bender, D.; Sartipi, K. (2013). \"HL7 FHIR: An Agile and RESTful approach to healthcare information exchange\". Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems: 326\u201331. doi:10.1109\/CBMS.2013.6627810.   \n\n\u2191 Kondylakis, H.; Petrakis, Y.; Leivadoros, S. et al. (2019). \"Using XDS and FHIR to Support Mobile Access to EHR Information Through Personal Health Apps\". Proceedings of the 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems: 241\u20134. doi:10.1109\/CBMS.2019.00058.   \n\n\u2191 Eckman, B.A.; Bennett, C.A.; Kaufman, J.H. et al. (2007). \"Varieties of interoperability in the transformation of the health-care information infrastructure\". IBM Systems Journal 46 (1): 19\u201341. doi:10.1147\/sj.461.0019.   \n\n\u2191 Nikoloudakis, Y.; Pallis, E.; Mastorakis, G. et al. (2019). \"Vulnerability assessment as a service for fog-centric ICT ecosystems: A healthcare use case\". Peer-to-Peer Networking and Applications 12: 1216\u201324. doi:10.1007\/s12083-019-0716-y.   \n\n\u2191 Nofer, M.; Gomber, P.; Hinz, O. (2017). \"Blockchain\". Business & Information Systems Engineering 59: 183\u20137. doi:10.1007\/s12599-017-0467-3.   \n\n\u2191 Vergne, J.P. (2020). \"Decentralized vs. Distributed Organization: Blockchain, Machine Learning and the Future of the Digital Platform\". Organization Theory 1 (4). doi:10.1177\/2631787720977052.   \n\n\u2191 Macdonald, M.; Liu-Thorrold, L.; Julien, R. (February 2017). \"The Blockchain: A Comparison of Platforms and Their Uses Beyond Bitcoin\". ResearchGate. doi:10.13140\/RG.2.2.23274.52164. https:\/\/www.researchgate.net\/publication\/313249614_The_Blockchain_A_Comparison_of_Platforms_and_Their_Uses_Beyond_Bitcoin .   \n\n\u2191 Kuo, T.-T.; Rojas, H.Z.; Ohno-Machado, L. (2019). \"Comparison of blockchain platforms: A systematic review and healthcare examples\". JAMIA 26 (5): 462\u201378. doi:10.1093\/jamia\/ocy185. PMC PMC7787359. PMID 30907419. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7787359 .   \n\n\u2191 Chowdhury, M.J.M.; Ferdous, S.; Biswas, K. et al. (2019). \"A Comparative Analysis of Distributed Ledger Technology Platforms\". IEEE Access 7: 167930\u201343. doi:10.1109\/ACCESS.2019.2953729.   \n\n\u2191 Triantafyllou, A.; Jimenez, J.A.P.; Torres, A.D.R. et al. (2020). \"The Challenges of Privacy and Access Control as Key Perspectives for the Future Electric Smart Grid\". IEEE Open Journal of the Communications Society 1: 1934\u201360. doi:10.1109\/OJCOMS.2020.3037517.   \n\n\u2191 Grammatikis, P.R.; Sarigiannidis, P.; Efstathopoulos, G. et al. (2020). \"ARIES: A Novel Multivariate Intrusion Detection System for Smart Grid\". Sensors 20 (18): 5305. doi:10.3390\/s20185305. PMC PMC7570496. PMID 32948064. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7570496 .   \n\n\u2191 Chockler, G.V.; Keidar, I.; Vitenberg, R. (2001). \"Group communication specifications: A comprehensive study\". ACM Computing Surveys 33 (4): 427\u201369. doi:10.1145\/503112.503113.   \n\n\u2191 35.0 35.1 Aguilera, M.K.; Keidar, I.; Malkhi, D. et al. (2011). \"Dynamic atomic storage without consensus\". Journal of the ACM 58 (2): 1\u201332. doi:10.1145\/1944345.1944348.   \n\n\u2191 Coiera, E.; Clarke, R. (2004). \"e-Consent: The design and implementation of consumer consent mechanisms in an electronic environment\". JAMIA 11 (2): 129\u201340. doi:10.1197\/jamia.M1480. PMC PMC353020. PMID 14662803. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC353020 .   \n\n\u2191 Bergmann, J.; Bott, O.J.; Hoffmann, I. et al. (2005). \"An eConsent-based System Architecture Supporting Cooperation in Integrated Healthcare Networks\". Studies in Health Technology and Informatics 116: 961\u20136. PMID 16160382.   \n\n\u2191 Bergmann, J.; Bott, O.J.; Pretschner, D.P. et al. (2007). \"An e-consent-based shared EHR system architecture for integrated healthcare networks\". International Journal of Medical Informatics 2\u20133: 130\u20136. doi:10.1016\/j.ijmedinf.2006.07.013. PMID 16971171.   \n\n\u2191 39.0 39.1 39.2 Staderini, M.; Schiavone, E.; Bondavalli, A. (2018). \"A Requirements-Driven Methodology for the Proper Selection and Configuration of Blockchains\". Proceedings of the 2018 IEEE 37th Symposium on Reliable Distributed Systems: 201\u20136. doi:10.1109\/SRDS.2018.00031.   \n\n\u2191 Markakis, E.; Nikoloudakis, Y.; Pallis, E. et al. (2019). \"Security Assessment as a Service Cross-Layered System for the Adoption of Digital, Personalised and Trusted Healthcare\". Proceedings of the 2019 IEEE 5th World Forum on Internet of Things: 91\u20134. doi:10.1109\/WF-IoT.2019.8767249.   \n\n\u2191 41.0 41.1 Angraal, S.; Krumholz, H.M.; Schulz, W.L. (2017). \"Blockchain Technology: Applications in Health Care\". Circulation: Cardiovascular Quality and Outcomes 10 (9): e003800. doi:10.1161\/CIRCOUTCOMES.117.003800. PMID 28912202.   \n\n\u2191 European Commission (29 July 2015). \"Commission Decision (EU) 2015\/1302 of 28 July 2015 on the identification of \u2018Integrating the Healthcare Enterprise\u2019 profiles for referencing in public procurement (Text with EEA relevance)\". EUR-Lex. https:\/\/eur-lex.europa.eu\/eli\/dec\/2015\/1302\/oj .   \n\n\u2191 Spanakis, E.G.; Spanakis, M.; Karantanas, A. et al. (2016). \"Secure access to patient's health records using SpeechXRays a mutli-channel biometrics platform for user authentication\". Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society: 2541-2544. doi:10.1109\/EMBC.2016.7591248. PMID 28268840.   \n\n\u2191 Grammatikis, P.I.R.; Sarigiannidis, P.G.; Moscholios, I.D. (2019). \"Securing the Internet of Things: Challenges, threats and solutions\". Internet of Things 5: 41\u201370. doi:10.1016\/j.iot.2018.11.003.   \n\n\u2191 Pasquale, F.A.; Rgone, T.A. (2014). \"Protecting Health Privacy in an Era of Big Data Processing and Cloud Computing\". Digital Commons. University of Maryland Francis King Carey School of Law. https:\/\/digitalcommons.law.umaryland.edu\/fac_pubs\/1542\/ .   \n\n\u2191 Linn, L.A.; Koo, M.B. (2016). \"Blockchain for health data and its potential use in health it and health care related research\" (PDF). Office of the National Coordinator for Health Information Technology. https:\/\/www.healthit.gov\/sites\/default\/files\/11-74-ablockchainforhealthcare.pdf .   \n\n\u2191 Miyachi, K.; Mackey, T.K. (2021). \"hOCBS: A privacy-preserving blockchain framework for healthcare data leveraging an on-chain and off-chain system design\". Information Processing & Management 58 (3): 102535. doi:10.1016\/j.ipm.2021.102535.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\">https:\/\/www.limswiki.org\/index.php\/Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on cybersecurityLIMSwiki journal articles on data management and sharingLIMSwiki journal articles on data regulationsLIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 8 June 2021, at 21:42.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 482 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","edf532278ba56a75b0d284a94702804c_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Emerging_and_established_trends_to_support_secure_health_information_exchange skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Emerging and established trends to support secure health information exchange<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>This work aims to provide <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, guidelines, established practices and standards, and an extensive evaluation on new and promising technologies for the implementation of a secure <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_sharing\" title=\"Data sharing\" class=\"wiki-link\" data-key=\"a99d5fda27f755c693c65864d9286130\">information sharing<\/a> platform for health-related data. We focus strictly on the technical aspects and specifically on the sharing of health information, studying innovative techniques for secure information sharing within the healthcare domain, and we describe our solution and evaluate the use of <a href=\"https:\/\/www.limswiki.org\/index.php\/Blockchain\" title=\"Blockchain\" class=\"wiki-link\" data-key=\"ae8b186c311716aca561aaee91944f8e\">blockchain<\/a> methodologically for integrating within our implementation. To do so, we analyze health information sharing within the concept of the PANACEA project, which facilitates the design, implementation, and deployment of a relevant platform. The research presented in this paper provides evidence and argumentation toward advanced and novel implementation strategies for a state-of-the-art information sharing environment; a description of high-level requirements for the national and cross-border transfer of data between different healthcare organizations; technologies to support the secure interconnectivity and trust between information technology (IT) systems participating in a data-sharing \u201ccommunity\u201d; standards, guidelines, and interoperability specifications for implementing a common understanding and integration in the sharing of clinical information; and the use of <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud computing<\/a> and prospectively more advanced technologies such as blockchain. The technologies described and the possible implementation approaches are presented in the design of an innovative secure information sharing platform in the healthcare domain.\n<\/p><p><b>Keywords<\/b>: interoperability, health information exchange, eHealth, blockchain, security, patient consent\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Information technology (IT) has long been identified as a cornerstone for efficient, cost-saving, timely, and reliable healthcare delivery.<sup id=\"rdp-ebb-cite_ref-CongressBringing95_1-0\" class=\"reference\"><a href=\"#cite_note-CongressBringing95-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KolodnerHealth08_2-0\" class=\"reference\"><a href=\"#cite_note-KolodnerHealth08-2\">[2]<\/a><\/sup> The availability of healthcare <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> and patient records in digital form facilitates the persistence and posterity of valuable information and greatly supports the decision-making process, and even the extraction of new knowledge at both the individual and population levels. In our previous work, we have emphasized the current state of the art about <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a> in the healthcare domain, with emphasis on current threats and methodologies.<sup id=\"rdp-ebb-cite_ref-SpanakisCyber20_3-0\" class=\"reference\"><a href=\"#cite_note-SpanakisCyber20-3\">[3]<\/a><\/sup> Paraphrasing the famous words of John Donne, \u201cno IT system is an island, entire of itself.\u201d Today, in a highly connected world where geographic boundaries have been largely eliminated and people can freely move between cities, states, countries, or continents, the requirement for two different <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">information management<\/a> systems to exchange a person's clinical data or medical history becomes vital and persistent. Sharing health information (e.g., via <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_information_exchange\" title=\"Health information exchange\" class=\"wiki-link\" data-key=\"7ea2eaa504cfa533dd9340169625c4ba\">health information exchange<\/a> [HIE]) through electronic means greatly improves the cost, quality, and patient experience of healthcare delivery.\n<\/p><p>To better secure the IT system's potential for interconnectivity and cooperation with other systems, the use of interoperable technologies and standards is needed. Depending on the extent and scope of the envisaged shared information spaces, there may be different levels of interoperability. Figure 1 shows a proposed \u201cmaturity\u201d model for interoperability in <a href=\"https:\/\/www.limswiki.org\/index.php\/EHealth\" title=\"EHealth\" class=\"wiki-link\" data-key=\"39df6aac1fbe4ad737280794f3a81d80\">eHealth<\/a>.<sup id=\"rdp-ebb-cite_ref-VanVelsenAMaturity16_4-0\" class=\"reference\"><a href=\"#cite_note-VanVelsenAMaturity16-4\">[4]<\/a><\/sup> The model consists of five levels that, incrementally, describe a more mature version of an interoperable infrastructure, starting from Level 1 for non-connected eHealth applications; Level 2 where a single eHealth application is directly linked to another application for simple data exchange<sup id=\"rdp-ebb-cite_ref-ChronakiAneHealth03_5-0\" class=\"reference\"><a href=\"#cite_note-ChronakiAneHealth03-5\">[5]<\/a><\/sup>; Level 3 for distributed systems that agree on protocols used, data formats, message exchange patterns, etc.<sup id=\"rdp-ebb-cite_ref-SpanakisR.26D05_6-0\" class=\"reference\"><a href=\"#cite_note-SpanakisR.26D05-6\">[6]<\/a><\/sup>; Level 4, where eHealth applications from different suppliers that serve a common goal are linked but the applications do not need to have common objectives<sup id=\"rdp-ebb-cite_ref-TsiknakisAdopt10_7-0\" class=\"reference\"><a href=\"#cite_note-TsiknakisAdopt10-7\">[7]<\/a><\/sup>; and finally, at the \u201cuniversal\u201d Level 5, where diverse eHealth applications connect to an open, interoperable infrastructure possibly spanning multiple countries.<sup id=\"rdp-ebb-cite_ref-SpanakisCongest18_8-0\" class=\"reference\"><a href=\"#cite_note-SpanakisCongest18-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SpanakisPharm19_9-0\" class=\"reference\"><a href=\"#cite_note-SpanakisPharm19-9\">[9]<\/a><\/sup>\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"0e54ecb91e83ee65d75c8c2245c8b358\"><img alt=\"Fig1 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/69\/Fig1_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 1.<\/b> A maturity model for interoperability in eHealth, adapted from Van Velsen <i>et al.<\/i> 2016<sup id=\"rdp-ebb-cite_ref-VanVelsenAMaturity16_4-1\" class=\"reference\"><a href=\"#cite_note-VanVelsenAMaturity16-4\">[4]<\/a><\/sup>.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Interoperability and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_sharing\" title=\"Data sharing\" class=\"wiki-link\" data-key=\"a99d5fda27f755c693c65864d9286130\">data sharing<\/a> in the healthcare domain is additionally challenging due to the multiplicity of the stakeholders, that is, the entities that operate (or are involved in any way) in this domain and which will be affected by any \u201cdisruption\u201d or reform of the system. Some of the most important stakeholders or actors are therefore the following:\n<\/p>\n<ul><li> the patients who are actually treated or, in general, are the recipients of the health services;<\/li>\n<li> the medical professionals (physicians and medical personnel) to provide the medical care;<\/li>\n<li> the healthcare organizations (HCOs) (healthcare providers) as represented by their board of directors, who actually administer the health delivery from a business perspective;<\/li>\n<li> the insurance companies that provide health coverage plans;<\/li>\n<li> the pharmaceutical companies that produce and market medications to be prescribed by physicians for the treatment of patients; and<\/li>\n<li> the governments and other regulatory parties who control, coordinate, and set the rules, rights, and obligations of any involved party.<\/li><\/ul>\n<p>All these actors could have an influence in the design of a data sharing system and can also set important\u2014and conflicting in some cases\u2014requirements. For example:\n<\/p>\n<ul><li> Patients would like to have their medical record shared, but only after their approval and only with specific authorized personnel in specific circumstances.<\/li>\n<li> An HCO can be extremely cautious about sharing the data of their patients with another organization because they are concerned by the security and availability of their systems.<\/li>\n<li> Governments of E.U. member states can impose strict laws about the transfer of their citizens in cross-border healthcare treatment scenarios.<\/li>\n<li> Medical professionals require fast and effortless access to a patient's medical history in emergency situations, which cannot be the case if time-consuming authorization processes are the norm. <\/li><\/ul>\n<p>Given these and similar requirements, and even though the objective is to design a technical solution for the sharing of clinical data, it is imperative that all these constraints and requirements are considered and addressed in a satisfying manner.\n<\/p><p>From a strictly technical point of view, the sharing platform may need to be interoperable with a large number and diverse set of IT systems, each with their own protocols, data formats, etc. Some of the most important systems that manage patient-related data, and could be used as data sources for information sharing, include <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a> (EHRs), <a href=\"https:\/\/www.limswiki.org\/index.php\/Personal_health_record\" title=\"Personal health record\" class=\"wiki-link\" data-key=\"f79b1a737e567e2fcb4eb85a4835d853\">personal health records<\/a> (PHRs), <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information systems<\/a> (LISs), and <a href=\"https:\/\/www.limswiki.org\/index.php\/Picture_archiving_and_communication_system\" title=\"Picture archiving and communication system\" class=\"wiki-link\" data-key=\"523b73ff51fa83663dc0b1d59e6d0f05\">picture archiving and communication systems<\/a> (PACS). EHRs are patient-centered systems that store and manage clinical information, such as a patient's medical history, diagnoses, medications, immunization dates, allergies, radiology images, and lab and test results. They are managed by authorized personnel, usually in the context of a single HCO, although they can span beyond that. PHRs are electronic applications that are used by people managing their own health information in a private and confidential environment. They are simpler systems than EHRs, and in some cases, they can be connected (temporarily or otherwise) to more enterprise-level HCO systems (e.g., EHRs or other <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital_information_system\" title=\"Hospital information system\" class=\"wiki-link\" data-key=\"d8385de7b1f39a39d793f8ce349b448d\">hospital information systems<\/a>). An LIS is used inside hospitals and clinics to record, manage, and store data for <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_laboratory\" title=\"Clinical laboratory\" class=\"wiki-link\" data-key=\"307bcdf1bdbcd1bb167cee435b7a5463\">clinical laboratories<\/a> in a patient-centric way (e.g., sending laboratory test orders to lab instruments, tracking those orders, and then recording the results in a searchable database). And PACS are systems used in a clinical setting for the storage and convenient access to medical images from multiple modalities (source machine types). Digital Imaging and Communications in Medicine (<a href=\"https:\/\/www.limswiki.org\/index.php\/DICOM\" title=\"DICOM\" class=\"wiki-link\" data-key=\"f0c7c747895286ff8785b6ed4dbc7ec0\">DICOM<\/a>) is the standard format and suite of protocols for the storage and transfer of images from PACS services. Of course, in a health ecosystem there may be additional systems, for example, for the management of insurance claims and for clinidal diagnoses (e.g., <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support systems<\/a> [CDSSs]).\n<\/p><p>In the general context, information sharing involves more than one party (healthcare providers, organizations, etc.,) that need to cooperate and agree on the way the exchange of information happens, and what the rules and policies are that govern it. Interoperability involves many different aspects, such as legislation and guidelines, contracts, and agreements between exchanging parties, governance and maintenance, shareable workflows, standardized data elements, semantic and syntactic choices, applications, technical infrastructure, and safety and privacy issues. The Refined eHealth European Interoperability Framework (EIF) is a set of recommendations that specify the standards, protocols, procedures, and policies that when deployed can improve the interoperability of eHealth applications within the E.U. and across its member states by providing specific recommendations for all these aspects.<sup id=\"rdp-ebb-cite_ref-EUeHealth15_10-0\" class=\"reference\"><a href=\"#cite_note-EUeHealth15-10\">[10]<\/a><\/sup> Figure 2 depicts how these aspects can be represented in interoperability \u201clevels\u201d that permit two different organizations to communicate.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"c0f86d3a6a8f3b3e1f24c782e390fd4b\"><img alt=\"Fig2 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/32\/Fig2_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 2.<\/b> Alignment of interoperability levels between two communicating organizations.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>This framework provides a great overview of the needed \u201cglue\u201d so that two or more healthcare environments can collaborate and serve as a common, multilevel, and multi-perspective model on the interoperability requirements. The six different levels of the (refined) EIF are:\n<\/p>\n<ul><li> Legal and regulatory: This level represents the legislation and regulatory guidelines that define the boundaries for interoperability not only across borders, but also within a country or region.<\/li>\n<li> Policy: This level represents the contracts and agreements between the sharing organizations so that trust is established and responsibilities are assigned.<\/li>\n<li> Care process: This level represents the shared workflows that define how the integrated care is delivered and how these workflows are managed.<\/li>\n<li> Information: This level defines the data models, the concepts and their values, the terminologies, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Controlled_vocabulary\" title=\"Controlled vocabulary\" class=\"wiki-link\" data-key=\"3df5d6c2e7def6be27a87de6fee0b844\">controlled vocabularies<\/a> that cater for the common understanding of the exchanged electronic messages.<\/li>\n<li> Applications: This level defines how the data are extracted from and imported to the healthcare information systems and how the transport of the data takes place using health-specific technologies and standards.<\/li>\n<li> IT infrastructure: This level represents the lowest level and corresponds to general-purpose communication and network protocols.<\/li><\/ul>\n<p>This generic EIF is highly relevant for the use cases of the health information sharing since sharing is greatly facilitated between interoperable systems and organizations. Here, we focus on the sharing of health-related information across HCOs and even across countries and continents. This is an important use case to improve the secure and efficient delivery of health care across Europe.<sup id=\"rdp-ebb-cite_ref-KatehakisSecurity17_11-0\" class=\"reference\"><a href=\"#cite_note-KatehakisSecurity17-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TinholtStudy13_12-0\" class=\"reference\"><a href=\"#cite_note-TinholtStudy13-12\">[12]<\/a><\/sup> The importance of cross-border health care in Europe has been recognized since 2011's Directive 2011\/24\/EU, which established patients' rights to access safe and high-quality healthcare, including across national borders within the E.U., and their right to be reimbursed for such healthcare.<sup id=\"rdp-ebb-cite_ref-PeetersFree12_13-0\" class=\"reference\"><a href=\"#cite_note-PeetersFree12-13\">[13]<\/a><\/sup> As can be deduced by considering the Refined eHealth EIF (Figure 2), the cross-border sharing of clinical information is a complex scenario due to the fact that data need to be transferred between different countries and therefore, requires overcoming barriers such as establishing a common trust framework, uniquely identifying citizens, and translating between different schemas and terminologies. This paper presents current approaches to address most of these issues in the European context while presenting and evaluating emerging technologies such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Blockchain\" title=\"Blockchain\" class=\"wiki-link\" data-key=\"ae8b186c311716aca561aaee91944f8e\">blockchain<\/a> that have been in the limelight recently. Our objective is to take advantage of both well-established and novel technologies that complement each other in order to design an architecture for the secure exchange of clinical information in the European context.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Current_status_on_standards-based_health_data_exchange\">Current status on standards-based health data exchange<\/span><\/h3>\n<p>In the healthcare industry, large standards developing organizations have defined numerous standards, data formats, terminologies, and more in order to support the design and building of interoperable IT systems. Perhaps the most well-known and most important standards are the ones introduced by <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"e0bf845fb58d2bae05a846b47629e86f\">Health Level 7<\/a> (HL7) and <a href=\"https:\/\/www.limswiki.org\/index.php\/SNOMED_CT\" title=\"SNOMED CT\" class=\"wiki-link\" data-key=\"a04000c818ae954bf52cfea5efdf020d\">SNOMED CT<\/a>, which can be used as a foundation for the development of data exchange standards among eHealth systems.<sup id=\"rdp-ebb-cite_ref-BensonPrinc12_14-0\" class=\"reference\"><a href=\"#cite_note-BensonPrinc12-14\">[14]<\/a><\/sup> Two more standards organizations are Integrating the Health Enterprise (IHE), which focuses primarily on integration and interoperability, and the <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_Data_Interchange_Standards_Consortium\" title=\"Clinical Data Interchange Standards Consortium\" class=\"wiki-link\" data-key=\"d5632c7df55eba08b01df1c257a3b8c9\">Clinical Data Interchange Standards Consortium<\/a> (CDISC). CDISC produced the Operational Data Model (ODM) to \u201cfacilitate the regulatory-compliant acquisition, archive, and interchange of <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a> and data for clinical research studies.\u201d<sup id=\"rdp-ebb-cite_ref-KuchinkeCDISC09_15-0\" class=\"reference\"><a href=\"#cite_note-KuchinkeCDISC09-15\">[15]<\/a><\/sup> ODM is an XML-based format that provides a number of constructs for modeling electronic <a href=\"https:\/\/www.limswiki.org\/index.php\/Case_report_form\" title=\"Case report form\" class=\"wiki-link\" data-key=\"9c8dc59f43e22ae67f769b59a1fdfe6f\">case report forms<\/a> (CRFs) and can also be used in sending forms data from a clinical trial system to an EHR system. In the area of medical devices, the Continua Health Alliance, a non-profit, open-industry coalition of healthcare and technology companies working to establish a system of interoperable personal health solutions, develops an ecosystem of connected technologies, devices, and services that will enable the more efficient exchange of fitness, health, and wellness information.<sup id=\"rdp-ebb-cite_ref-CarrollContinua07_16-0\" class=\"reference\"><a href=\"#cite_note-CarrollContinua07-16\">[16]<\/a><\/sup> Among its proposed standards, Continua proposes specifications and standards such as Bluetooth, USB, <a href=\"https:\/\/www.limswiki.org\/index.php\/ISO\/IEEE_11073\" title=\"ISO\/IEEE 11073\" class=\"wiki-link\" data-key=\"f43ec7169a3f50e3151de0fdef5867cc\">ISO\/IEEE 11073<\/a> (for medical devices), and HL7 to enable patients to use home-based devices to monitor their weight, blood pressure, and glucose and blood oxygen levels, as well as to share these data with their healthcare professionals.\n<\/p><p>The exchanged information can be in multiple data formats based on the type of data, device category, etc.<sup id=\"rdp-ebb-cite_ref-MariasClin09_17-0\" class=\"reference\"><a href=\"#cite_note-MariasClin09-17\">[17]<\/a><\/sup> Some of the most common formats based on the health applications using them are the following:\n<\/p>\n<ul><li> For medical imaging, the use of DICOM is almost universal and defines not only the content (DICOM file format) but also communication protocols for the exchange of medical images.<sup id=\"rdp-ebb-cite_ref-LarobinaMedical13_18-0\" class=\"reference\"><a href=\"#cite_note-LarobinaMedical13-18\">[18]<\/a><\/sup> <\/li>\n<li> In the area of DNA sequencing and other -omics data formats, we have FASTQ file format<sup id=\"rdp-ebb-cite_ref-CockTheSanger10_19-0\" class=\"reference\"><a href=\"#cite_note-CockTheSanger10-19\">[19]<\/a><\/sup>, which is used to store sequence information, and the standard flowgram format (SFF), which is used to encode sequence reads.<sup id=\"rdp-ebb-cite_ref-LeinonenTheSeq11_20-0\" class=\"reference\"><a href=\"#cite_note-LeinonenTheSeq11-20\">[20]<\/a><\/sup><\/li>\n<li> The majority of the EHR systems adopt the HL7 standard clinical document architecture (CDA) as the interoperable data format.<sup id=\"rdp-ebb-cite_ref-DolinTheHL7_01_21-0\" class=\"reference\"><a href=\"#cite_note-DolinTheHL7_01-21\">[21]<\/a><\/sup> CDA is part of the HL7 Version 3 family, and it is based on a reference information model (RIM) that serves as a semantic model that consists of a set of structural components (e.g., classes with data types) and semantic relations that are used to represent clinical notes in the form of an extensible markup language document.<\/li><\/ul>\n<p>The use of controlled vocabularies and terminologies allows for the unambiguous representation of important value sets such as \"diagnosis\" and \"prescribed medicines.\"<sup id=\"rdp-ebb-cite_ref-Freitas_22-0\" class=\"reference\"><a href=\"#cite_note-Freitas-22\">[22]<\/a><\/sup> The following are examples of such terminologies:\n<\/p>\n<ul><li> The <a href=\"https:\/\/www.limswiki.org\/index.php\/International_Statistical_Classification_of_Diseases_and_Related_Health_Problems\" title=\"International Statistical Classification of Diseases and Related Health Problems\" class=\"wiki-link\" data-key=\"1de9af67005dfe2895e5d8cf6de57d4a\">International Classification of Diseases<\/a> (ICD) provides a common language for reporting and monitoring diseases, used throughout the world to compare and share data in a consistent standard way between hospitals, regions, and countries and over periods of time. It is used to classify diseases and other problems for payment, management, and research, as recorded on many types of health records including medical records and death certificates. ICD-11 is the latest version of it, whereas ICD-10 (released in 1993) remains widely used.<\/li>\n<li> SNOMED CT, already mentioned above, is the most comprehensive multilingual clinical healthcare terminology available. It is used in EHR systems to facilitate clinical documentation and reporting and to retrieve and analyze clinical data. SNOMED CT is both a coding scheme, identifying concepts and terms, and a multidimensional classification, enabling concepts to be related to each other, grouped, and analyzed according to different criteria.<\/li>\n<li> Logical Observation Identifiers Names and Codes (<a href=\"https:\/\/www.limswiki.org\/index.php\/LOINC\" title=\"LOINC\" class=\"wiki-link\" data-key=\"b20c83dbfab36194bab6c223e31ebfdc\">LOINC<\/a>) provides a set of universal identifiers for medical laboratory observations. LOINC provides codes for the observation names (e.g., eye color), not the observation finding (e.g., blue eyes). LOINC therefore provides codes for questions, and where needed, other vocabularies, such as SNOMED CT, provide codes for the answers.<\/li>\n<li> The <a href=\"https:\/\/www.limswiki.org\/index.php\/Unified_Medical_Language_System\" title=\"Unified Medical Language System\" class=\"wiki-link\" data-key=\"cdee33b213990e5416c0b7fba322ce7c\">Unified Medical Language System<\/a> (UMLS) is an important terminology resource, intended for use mainly by developers of health information systems. The UMLS \u201cMetathesaurus\u201d uses several different source vocabularies and seeks to reflect and preserve the meanings of concept names and relationships from these sources. It is therefore a valuable resource for the translation between the different source vocabularies.<\/li><\/ul>\n<p>Application-level interfaces are also needed to support the communication and exchange of the standards-based encoded information. The role of HL7 is principal on this front: HL7's name comes from \u201cLevel Seven,\u201d which, according to the Open Systems Interconnection (OSI) model that standardizes communication functionality in IT, corresponds to application layer. From its establishment in the late 1980s, HL7 was therefore focused on exchanging information within hospitals. The focus remains almost the same today, but HL7 has progressed from different paradigms over the years, in order to describe the structure, semantics, and management of the exchanged information. The development of HL7 Version 3 (HL7v3) started around 1995 in order to introduce more consistency between the implementations of Version 2 following an object-oriented development methodology. The most recent proposal by HL7 is <a href=\"https:\/\/www.limswiki.org\/index.php\/Fast_Healthcare_Interoperability_Resources\" title=\"Fast Healthcare Interoperability Resources\" class=\"wiki-link\" data-key=\"65dd2a848285f9151006e17e036d596d\">Fast Healthcare Interoperability Resources<\/a> (FHIR), which leverages web technologies to overcome the complexity of HL7v3.<sup id=\"rdp-ebb-cite_ref-BenderHL713_23-0\" class=\"reference\"><a href=\"#cite_note-BenderHL713-23\">[23]<\/a><\/sup>\n<\/p><p>On the other hand, the IHE initiative has defined a number of \u201cintegration profiles,\u201d which are detailed specifications for communication among systems to address key clinical use cases, all based on established standards. IHE profiles organize and leverage the integration capabilities that can be achieved by coordinated implementation of communication standards, such as DICOM, HL7, W3C, and other security standards.<sup id=\"rdp-ebb-cite_ref-KondylakisUsing19_24-0\" class=\"reference\"><a href=\"#cite_note-KondylakisUsing19-24\">[24]<\/a><\/sup> Some of the IHE integration profiles that might be interesting in the context of interfacing health information systems and sharing of clinical information include:\n<\/p>\n<ul><li> Cross-Enterprise Document Sharing (XDS): Allows EHR documents to be shared and discovered among healthcare enterprises, physician offices, clinics, acute care in-patient facilities, and PHRs.<\/li>\n<li> Patient Demographics Query (PDQ): Enables applications to query by patient demographics (e.g., name) for patient identity from a central patient information server.<\/li>\n<li> Patient Identifier Cross Referencing (PIX): Allows applications to query for patient identity cross-references between hospitals, sites, HIE networks, etc.<\/li>\n<li> PDQ HL7 v3 (PDQv3): Extends the PDQ profile leveraging HL7 Version 3.<\/li>\n<li> PIX: Extends the Patient Identifier Cross-Reference profile leveraging HL7 Version 3.<\/li>\n<li> Cross-Community Access (XCA): Allows the query and retrieval of patients' EHRs held by other communities.<\/li>\n<li> Cross-Enterprise Document Reliable Interchange (XDR): Exchanges health documents between health enterprises using a web service-based point-to-point push network communication.<\/li><\/ul>\n<p>There are two main architectural approaches for the implementation of an information sharing platform: a centralized approach and a federated approach.<sup id=\"rdp-ebb-cite_ref-EckmanVarieties07_25-0\" class=\"reference\"><a href=\"#cite_note-EckmanVarieties07-25\">[25]<\/a><\/sup> In the centralized approach, a central data warehouse and accompanying services act as middlemen for the exchange of information and a single source of patient data that are shared among the participating organizations. On the other hand, in the federated architecture, a central infrastructure is also in place, but in this case, it merely acts as a facilitator for locating the data sources. An example of this case would be a common registry that stores only the links to the original patient records, medical images, and other types of data while the linked data are not transferred outside their primary premises unless explicitly requested by any interested client system. In addition to these opposite approaches for designing a distributed information sharing platform, there are also various hybrid options, such as using messaging with \u201cpublish-subscribe\u201d communication that can be introduced to complement either the centralized or federated architectures.\n<\/p><p>There are advantages and disadvantages in all of the abovementioned deployment options. For example, in the federated approach, there are more strong concerns about the privacy, security, and availability of the data shared and their original sources.<sup id=\"rdp-ebb-cite_ref-NikoloudakisVulner19_26-0\" class=\"reference\"><a href=\"#cite_note-NikoloudakisVulner19-26\">[26]<\/a><\/sup> The operation of a mission-critical <a href=\"https:\/\/www.limswiki.org\/index.php\/Radiology_information_system\" title=\"Radiology information system\" class=\"wiki-link\" data-key=\"62d0a667f8b27309ea38a46507cfb500\">radiology information system<\/a> (RIS) in a hospital can be severely affected if multiple peers request DICOM images from its PACS, and this poses an additional burden and cost for the acquisition and management of adequate infrastructure in the source organization. Instead, a centralized strategy allows for easy access to the whole information shared but also leads to a concentration of the costs for maintaining the infrastructure needed and can be problematic at the operation level (a \u201csingle point of failure\u201d). Furthermore, there are more costs on integrating the different data sets under a common \u201cschema,\u201d resolving conflicts or even supporting the timely update of the persisted information when a source system acquires new or modified data.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Emerging_supportive_technologies:_Blockchain\">Emerging supportive technologies: Blockchain<\/span><\/h3>\n<p>Blockchain is a decentralized, distributed data structure used to store transactions (aggregated in blocks) across many computers.<sup id=\"rdp-ebb-cite_ref-NoferBlock17_27-0\" class=\"reference\"><a href=\"#cite_note-NoferBlock17-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VergneDecentral20_28-0\" class=\"reference\"><a href=\"#cite_note-VergneDecentral20-28\">[28]<\/a><\/sup> Blockchain has been extensively used for Bitcoin.<sup id=\"rdp-ebb-cite_ref-MacdonaldTheBlock17_29-0\" class=\"reference\"><a href=\"#cite_note-MacdonaldTheBlock17-29\">[29]<\/a><\/sup> In the realm of healthcare, we have seen Kuo <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-KuoComp19_30-0\" class=\"reference\"><a href=\"#cite_note-KuoComp19-30\">[30]<\/a><\/sup> perform a systematic review on how blockchain can be used in healthcare applications. Blockchain core is the embedded distributed ledger technology able to support <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">data integrity<\/a>, authenticity, and origin. In blockchain, each block is linked to the previous one through a cryptographic hash, and each block is a data structure that allows the storage of a list of transactions.<sup id=\"rdp-ebb-cite_ref-ChowdhuryAComp19_31-0\" class=\"reference\"><a href=\"#cite_note-ChowdhuryAComp19-31\">[31]<\/a><\/sup> In the blockchain, a transaction abstracts and allows the tracking of an exchange or interaction between two entities. Transactions are created and exchanged by peers of the blockchain network and modify the state of the blockchain data structure. An efficient categorization and comprehensive overview of the latest privacy-preserving mechanisms and policies regarding smart electric grids, focusing on the use of the blockchain technology and the multi-authority access control paradigm, is offered by Triantafyllou <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TriantafyllouTheChall20_32-0\" class=\"reference\"><a href=\"#cite_note-TriantafyllouTheChall20-32\">[32]<\/a><\/sup> and Radoglou <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-GrammatikisARIES20_33-0\" class=\"reference\"><a href=\"#cite_note-GrammatikisARIES20-33\">[33]<\/a><\/sup>\n<\/p><p>Concerning data access, we can have the following:\n<\/p>\n<ul><li> Public blockchain: There are no restrictions on reading blockchain data and submitting transactions for inclusion into the blockchain.<\/li>\n<li> Private blockchain: Direct access to blockchain data and submitting transactions is limited to a predefined list of entities.<\/li><\/ul>\n<p>Concerning data management, we can have the following:\n<\/p>\n<ul><li> Permissioned blockchain: Transaction processing is performed by a predefined list of peers with known identities.<\/li>\n<li> Permissionless blockchain: No restrictions on identities of transaction processors (i.e., blocks creators).<\/li><\/ul>\n<p>Combining the two perspectives, we can have four categories, as depicted in Figure 3.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"2a0f40d2f22bb6383511d919f848cfd6\"><img alt=\"Fig3 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/92\/Fig3_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 3.<\/b> Blockchain classification overview.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><br \/>\nBlockchain supports auditability and transparency, as any reader is able to verify the correctness of the state of the system. Indeed, by storing all the transactions, it is possible to replay (starting from a correct checkpoint) the entire history and check that the current state is consistent with the set of recorded transactions. It is important to note that, when using a blockchain to store data, there exists an inherent tradeoff between transparency and privacy. Indeed, if the primary requirement is to have a fully transparent system, we need to accept that anyone is allowed to see any piece of information (sacrificing privacy). Conversely, if the primary requirement is to have a private system, it will not provide transparency. A tradeoff between transparency and privacy is however possible, but it will come at the cost of efficiency, as it would require employing complex cryptographic primitives.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"A_novel_view_on_information_sharing\">A novel view on information sharing<\/span><\/h3>\n<p>Nowadays, the sharing of a patient's clinical information between two HCOs (e.g., hospitals) usually requires a great amount of manual work in order to check and validate patient's consent (by consulting signed papers) or, at worst, results in privacy loss by extending the trust circle, e.g., to all physicians from the requesting organization. The main limitation of the current sharing pattern can be summarized in the following way:\n<\/p>\n<ol><li> Sharing medical data may require time and possibly multiple interactions, also involving the patient in the loop.<\/li>\n<li> Sharing medical data is currently a physical point-to-point interaction. If the same set of data needs to be shared with multiple parties, it would require multiple sharing patterns to be in place.<\/li>\n<li> Sharing is currently asymmetric. Organization A may have the consent to share patient's data with organization B, but the inverse may not be true.<\/li><\/ol>\n<p>In order to overcome these deficiencies, we aim to design a platform\u2014the Innovative Secure Information Sharing Platform (InSISP)\u2014that is able to support a fast and efficient medical information sharing at both national and cross-national levels, taking into account sharing constraints, including those imposed by the <a href=\"https:\/\/www.limswiki.org\/index.php\/General_Data_Protection_Regulation\" title=\"General Data Protection Regulation\" class=\"wiki-link\" data-key=\"3f4bdf6f0dcb360b1e79aad8674c2447\">General Data Protection Regulation<\/a> (GDPR). It is imperative that patient's consent is central in this framework, and one of the challenges is to make the consent management robust, simple, and secure. A sequence diagram of the possible interactions to support the data sharing is shown in Figure 4.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"de285db8341bf8a5918f27170a311d1e\"><img alt=\"Fig4 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/92\/Fig4_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 4.<\/b> InSISP sharing pattern for medical data.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>From an abstract point of view, InSISP can be seen as a data repository that can be accessed by HCOs (i.e., clients) to store and retrieve shared data by using a common interface and format (e.g., CDA Release 2 using standard vocabularies such as SNOMED and LOINC). The shared data repository is surrounded by a federation of collaborating entities (organizations\/clients) that is dynamic and evolving. Once the federation is established, participating entities can start sharing data according to the data processing consent provided by patients. To this aim, we can identify two additional functionalities: (i) data sharing and (ii) data processing consent management. Figure 5 summarizes a possible decomposition of the InSISP and highlights the three storage components.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"b1c735e740c5a233f6cfae51640061d4\"><img alt=\"Fig5 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a2\/Fig5_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 5.<\/b> InSISP functional decomposition.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"Federation_management\">Federation management<\/span><\/h4>\n<p>The federation management functionality has the aim to manage the federation life cycle, and in particular, it should allow new HCOs to join and HCOs no longer interested in participating in the federation to leave. In particular, this functionality supports the following operations:\n<\/p>\n<ol><li> Join the federation: HCOs should be able to become part of the federation at any time. When considering a membership service, all the members are assumed to be uniquely identified. So from the perspective of the InSISP development, there should be an external service that is able to support the identification of members and to provide them with a digital identity.<\/li>\n<li> Leave the federation: HCOs may decide to leave the federation at their will, and the InSISP should support the removal of the entity from the membership and should notify the end of the sharing to connected entities.<\/li>\n<li> Get the federation membership: Allows an HCO participating in the federation to get the current membership and know the set of HCOs potentially involved in the sharing, including their identification, public keys, and the categories of data that are currently available for the sharing.<\/li><\/ol>\n<p>Federation management intrinsically relies on the execution of a distributed protocol running, and thus, there are two main options to implement a federation membership service: (i) client\/server or (ii) peer to peer. In the client\/server case, the current membership of the system is maintained by a trusted third party (TTP). When a new HCO wants to join the federation, it simply needs to contact the TTP and identify and authenticate itself with the TTP that will proceed by adding it to the current view. Similarly, when an HCO in the federation wants to leave, it simply needs to notify the TTP that will remove it from the current view. The current membership can be obtained again by querying the TTP. The main advantage of this option is that all the complexities of the membership management are delegated to the TTP. However, this also implies that the TTP is clearly a single point of failure for the system as well as its main bottleneck. In the peer-to-peer approach, HCOs collaborate to maintain a consistent view of the system by exchanging messages and trying to reach a consensus on the sequence of views generated to include new members and to remove old ones.\n<\/p><p>Chockler <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ChocklerGroup01_34-0\" class=\"reference\"><a href=\"#cite_note-ChocklerGroup01-34\">[34]<\/a><\/sup> discussed in detail the formalization and the specification of the group membership service, while more recently, Aguilera <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-AguileraDynamic11_35-0\" class=\"reference\"><a href=\"#cite_note-AguileraDynamic11-35\">[35]<\/a><\/sup> considered the problem of building a reconfiguration service to support the development of a distributed shared storage. However, let us note that in all these cases, the emphasis is on how to provide a consistent view to all the members. To the best of our knowledge, there does not exist any approach investigating the cost of realizing a membership service using blockchain technologies.\n<\/p><p>The main advantage of the peer-to-peer approach is its intrinsic resilience. In addition, in peer-to-peer settings, it is also possible to consider a blockchain-based approach to construct the sequence of consistent views providing the \"view auditability\" property for free. The main drawback is the cost imposed by the management of consistent views, as it requires to run coordination and synchronization protocols among all the participants that would bring poor scalability in case of a highly dynamic federation.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Data_processing_consent_management\">Data processing consent management<\/span><\/h4>\n<p>The data processing consent management function has the aim to support the development of a digital data processing activity registry to store and access patients' data processing consent. The data processing consent is granted by a patient for a specific set of data to a specific set of entities and for a specific purpose and period of time. In order to support the automatic verification of patients' consent, such information must be stored and managed by the HCO in an electronic form (e-Consent).<sup id=\"rdp-ebb-cite_ref-CoieraEConsent04_36-0\" class=\"reference\"><a href=\"#cite_note-CoieraEConsent04-36\">[36]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BergmannEneConsent05_37-0\" class=\"reference\"><a href=\"#cite_note-BergmannEneConsent05-37\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BergmannAnEconsent07_38-0\" class=\"reference\"><a href=\"#cite_note-BergmannAnEconsent07-38\">[38]<\/a><\/sup> Also, every patient has the right to modify their consent and has the right to be forgotten; i.e., at any point in time, they may ask to revoke all their previous consent (while also erasing any data identifying them).\n<\/p><p>To this aim, the data processing consent management functionality should offer the following operations:\n<\/p>\n<ol><li> Provide new consent: It should allow the addition of a new entry to the table, specification of the beneficiary entities, and seting of the expiration time of the consent.<\/li>\n<li> Update consent: It should allow the modification of existing consent by allowing sharing with a new beneficiary or by removing a beneficiary.<\/li>\n<li> Remove (all) consent: It should basically implement the right to be forgotten by deleting all the consent previously provided by a specific patient.<\/li><\/ol>\n<p><b>Discussion about possible design and deployment options<\/b>\n<\/p><p>Let us note that the data processing consent management function supports every HCO in managing its own data processing activity registry. Thus, from this point of view, we can say that it is local to every HCO.\n<\/p><p>As a consequence, the most appropriate choice is to design it as a local data store managed and accessed only by one HCO. Of course, in order to increase the resiliency and security of the storage, it can be also replicated, but all the replicas will still be managed by the same HCO.\n<\/p><p>However, a distributed design raises a privacy issue in patients' information. Indeed, even if data in the registry are not sensible by themselves, they could be easily correlated to infer sensitive information about patients and would result in a privacy violation; e.g., by looking to the list of HCOs where Bob did his analysis, you may infer that Bob is affected by a specific disease. To solve this issue, it is necessary to employ anonymization scheme generation, an extra cost without any particular advantage in terms of reliability or security.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Data_sharing_management\">Data sharing management<\/span><\/h4>\n<p>Data sharing is the core functionality of the InSISP, as it manages the real transfer of medical data between parties. It offers just one main operation, i.e., Get Data, which is used to retrieve a specific piece of data for a specific patient and transfer it according with the patient's consent.\n<\/p><p>We can consider three main options to design and deploy this functionality: client\/server, peer-to-peer with message exchanges, and peer-to-peer with shared memories. In the client\/server case, data available for sharing are copied and pushed toward a centralized TTP that will take care of satisfying the sharing request. In order to be GDPR-compliant, a specific consent to move data to the TTP must be signed, as well as the consent to share data with all the federation members. (Let us note that this second set of consent could be removed if every HCO provides the TTP a copy of its data processing activity registry. However, as mentioned above, this would add the complexity of finding a good anonymization scheme that allows to preserve patients' privacy still allowing the TTP to check consent.) Figure 6 shows an overview of a possible client\/server design.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"42ff0b9078446c411dd9be120855c636\"><img alt=\"Fig6 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/fc\/Fig6_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 6.<\/b> Overview of the client\/server design with a trusted third party (TTP).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In the peer-to-peer case, the idea is that the sharing is realized by letting HCOs in the federation cooperate with each other. We can distinguish two cases: cooperation realized through message exchange and cooperation realized through shared memory.\n<\/p><p>In the message exchange case, the sharing is realized using an <i>ad hoc<\/i> request\u2013reply communication pattern, as shown in Figure 7.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"1d871e9799ff1adb8902f5205ef63a3f\"><img alt=\"Fig7 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b6\/Fig7_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 7.<\/b> Overview of the peer to peer with message exchange design.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In the peer to peer with shared memory design (Figure 8), each HCO creates a shared memory space where it stores all the pieces of data that can be shared according to the data processing activity registry. As an example, let us consider the case where Bob provided the consent to HCO<sub>1<\/sub> to share his X-ray images with HCO<sub>2<\/sub>. This means that HCO<sub>1<\/sub> and HCO<sub>2<\/sub> create locally a shared space where they will store a copy of all Bob's X-ray images (the red slice in Figure 8).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"47c61cd8a6814d3554ef9ed6fa9fa75c\"><img alt=\"Fig8 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/ad\/Fig8_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 8.<\/b> Overview of the peer to peer with shared memory design.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><b>Discussion about possible design and deployment options<\/b>\n<\/p><p>Technically, all the three considered designs are feasible. However, as anticipated in the previous section, the client\/server scenario poses several challenges from the point of view of the consent needed in order to make it compliant. In particular, the main issue is the large set of consent data that is necessary, and that patients may be reluctant to provide consent in the first place. Instead, the two point-to-point designs solve this issue, as they exploit locally the consent information and move data only toward authorized HCOs.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Summary_of_recommendations\">Summary of recommendations<\/span><\/h4>\n<p>According to the considerations discussed in the previous sections, Table 1 summarizes the viable options for the design of each functionality of the InSISP.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"15a167f569b29006d5e993a26ace7aff\"><img alt=\"Tab1 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a5\/Tab1_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Table 1.<\/b> Summary of design options and recommendations.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Evaluation_of_emerging_technical_solutions:_Blockchain\">Evaluation of emerging technical solutions: Blockchain<\/span><\/h3>\n<p>In order to evaluate if blockchain is a valid option to support the InSISP deployment, the methodology presented by Staderinin <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-StaderiniAReq18_39-0\" class=\"reference\"><a href=\"#cite_note-StaderiniAReq18-39\">[39]<\/a><\/sup> has us considering the following three steps:\n<\/p>\n<ol><li> Perform a requirements analysis to assess blockchain benefits.<\/li>\n<li> Evaluate the most appropriate blockchain solution where the designer is guided on the choice of the most suitable blockchain category, based on blockchain-specific criteria, depending on who are readers and writers of the data and who is allowed to generate data.<\/li>\n<li> Select the blockchain configuration, which assists the designer throughout the decision-making process for the configuration of the blockchain compliantly with the chosen category and the given project requirements.<\/li><\/ol>\n<p>Let us remark that blockchain is intrinsically a distributed system, and it makes no sense trying to use it when a distributed setting is not appropriate, which means that it can be used for the federation membership and for the data sharing functionalities of InSISP. In the following, we will evaluate the suitability of blockchain-based solution by adoption of the methodology described by Aguilera <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-AguileraDynamic11_35-1\" class=\"reference\"><a href=\"#cite_note-AguileraDynamic11-35\">[35]<\/a><\/sup>\n<\/p><p>In fact, trying to evaluate the blockchain technologies for the data sharing scenarios and design solutions described above, the first step is the analysis of requirements related to the component under analysis in order to understand the benefit of adopting a blockchain-based solution for its low-level design and development. The factors that are considered in this step include:\n<\/p>\n<ul><li> <i>Data or state storage<\/i>: The first element to consider is to check if the module under analysis needs to store data or system states. If no information needs to be stored, clearly no blockchain is needed.<\/li>\n<li> <i>Immutability and data integrity<\/i>: With immutability, we refer to the property of a piece of data to never change (i.e., a constant value that is never updated). If immutability is a requirement, then blockchain is certainly an option, as this is probably the most distinctive property of any blockchain. Integrity is strictly related to immutability, and this is why they are analyzed together, also considering that they are both closely related to cryptography. If a component requires data protection from unauthorized modifications, then this requirement can be met with blockchain.<\/li>\n<li> <i>Non-repudiation<\/i>: Non-repudiation means that the author of some message\/data cannot deny that it produced the message. This is another fundamental property that can be easily satisfied using blockchains.<\/li>\n<li> <i>Multiple writers<\/i>. This criterion considers the multiplicity of entities in charge of writing data in the storage. If only one entity is a writer, then a common database is probably more appropriate than a blockchain, especially from the performance perspective, i.e., in terms of throughput and latency.<\/li>\n<li> <i>TTP always online<\/i>: A TTP is an entity that facilitates interactions between mutually mistrusting entities. If in the system a TTP is required and it is planned to be always online, entities can delegate write operations to it as transactions, or as state changes. Therefore, the TTP plays the role of a trusted deliverer and verifier. In this case, a blockchain, known for being a trustless technology, becomes useless, and the methodology brings to the related output. Otherwise, it can happen that the involvement of a TTP is planned but not for being always online: in this case, it could play the role of an authority giving authorizations for permissioned blockchains. Alternatively, a TTP may not exist at all. In the latter situations, it is not possible to exclude the recommendation of using a blockchain.<\/li>\n<li> <i>Writers are known and trusted<\/i> If all the entities interested in writing know and mutually trust each other, a blockchain is superfluous and not recommended (again mainly for performance issues).<\/li><\/ul>\n<p>The flowchart of this analysis step is shown in Figure 9. Tables 2 and 3 then answer the relevant questions for the assessment of blockchain in the federation management and data sharing operations according to the decision flowchart in Figure 9. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig9_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"5edd6f4ec61317697b3eeb91fbc09c6a\"><img alt=\"Fig9 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/4b\/Fig9_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 9.<\/b> Flowchart: blockchain, yes or no? adapted from Staderini <i>et al.<\/i> 2018<sup id=\"rdp-ebb-cite_ref-StaderiniAReq18_39-1\" class=\"reference\"><a href=\"#cite_note-StaderiniAReq18-39\">[39]<\/a><\/sup>.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab2_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"c2994b73845201d89fac70797422ce7d\"><img alt=\"Tab2 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/ff\/Tab2_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Table 2.<\/b> Assessment of blockchain in the federation management.<sup id=\"rdp-ebb-cite_ref-StaderiniAReq18_39-2\" class=\"reference\"><a href=\"#cite_note-StaderiniAReq18-39\">[39]<\/a><\/sup>.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab3_Spanakis_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"c87ae65baae65453a8fa0972a1a4f57e\"><img alt=\"Tab3 Spanakis FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/47\/Tab3_Spanakis_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Table 3.<\/b> Assessment of blockchain for data sharing operations.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Going back to the maturity model flowchart in Figure 1, we at first glance may be led to believe that blockchain is not recommended to support the implementation of the federation membership mainly because there exists a basic level of trust between members of the federation. In addition, we may be led to assume that the federation membership is also relying on a trusted external service providing digital and secure identities to participants. However, we must highlight the opportunity to consider a non-repudiation requirement; as such, we considered the importance of preserving data integrity (i.e., to ensure that the current membership cannot be altered). Thus, if these two requirements become more relevant or if assumptions on the identity platform or about trustworthiness of participant cannot be met, then blockchain immediately becomes a viable solution.\n<\/p><p>Using the same process, at first, we are led to believe that blockchain is recommended to support the implementation of data sharing mainly because it would efficiently support the data integrity requirement, i.e., it allows the traceability of data accesses and verifies their authorship and integrity. However, we should keep in mind that medical data are mostly read-only, and in our context, they are shared between trusted parties. Thus, the main benefit we can get by adopting a blockchain-based solution is the support for data integrity verification and data auditability. Yet even still, this feature must be carefully balanced with the \u201cright to be forgotten\u201d requirement (i.e., a MUST requirement imposed by the GDPR regulation) and its implication on the adoption of a blockchain-based solution. In order to support the implementation of the \u201cright to be forgotten,\u201d we need to guarantee that data can be deleted from the blockchain when the data owner asks for it to be done. Currently, deleting data efficiently from a blockchain is still an open research problem, and the few existing solutions are currently based on the adoption of computationally expensive cryptographic techniques. Furthermore, when dealing with medical data, there is also the additional complexity following the huge heterogeneity of data to be considered (i.e., text, images, and images\/sounds). Blockchain technologies have been originally designed to deal with transactional data, of small size and in the form of numbers or strings. Currently, it is not clear how to extend the paradigm to work with heterogeneous data. A possible solution to this issue could be to keep such heterogeneous medical data stored locally in a classical database and store in the blockchain only its hash. However, it is still not clear how much privacy lawyers consider such metadata as an expression of a personal data, and thus, the issue may remain.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Information sharing in the health domain is a complex and challenging process given the many stakeholders involved, the different and sometimes competing standards and solutions to choose from, and the important security, ethics, and regulation-related constraints for any proposed solution to comply with any \"security as a service\" attempts, e.g., as outlined by Markakis <i>et al<\/i>.<sup id=\"rdp-ebb-cite_ref-MarkakisSecurity19_40-0\" class=\"reference\"><a href=\"#cite_note-MarkakisSecurity19-40\">[40]<\/a><\/sup> HIE is a key building block for the realization of Connected Health in Europe, which \u201cspeaks to the health journey of the person, through the entire lifespan, leveraging a variety of technologies to do so.\u201d<sup id=\"rdp-ebb-cite_ref-AngraalBlock17_41-0\" class=\"reference\"><a href=\"#cite_note-AngraalBlock17-41\">[41]<\/a><\/sup> Based on what we've discussed in this work, it is important to consider the following aspects when building a new platform for sharing medical information:\n<\/p>\n<ul><li> Patient consent is of utmost importance, and infrastructure should be in place for its registration, enforcement, and withdrawal.<\/li>\n<li> Compliance with GDPR and national laws is vital, as is implementing solutions to address significant requirements, such as the \u201cright to erasure.\u201d<\/li>\n<li> Effective interoperable solutions are produced by linking and integrating with well-established standards such as document and data formats (e.g., CDA and DICOM) and metadata and value lists (e.g., SNOMED and LOINC) in order to support common understanding and integration.<\/li>\n<li> Such solutions should be able to andle the whole security spectrum, including authentication and authorization of users, as well as support for data privacy, audits for \u201cpost-mortem\u201d analysis and non-repudiation, data integrity, and machine-enforced trust among the sharing organizations.<\/li>\n<li> Such solutions will also enable the unique identification of patients while at the same time exposing the minimal set of personal information in order to protect their privacy.<\/li>\n<li> Performance, scalability, and availability of the whole platform should be high in order to support the health-related processes efficiently.<\/li>\n<li> The solution should be part of the healthcare ecosystem, which means allowing easy integration with existing infrastructure by featuring interoperable \u201cports and adapters\u201d interfaces.<\/li><\/ul>\n<p>Additionally, the IHE profiles should not be neglected. In the 2015\/1302 Commission Decision, after consulting the European multistakeholder platform on information and communications technology (ICT) standardization and sectoral experts, 27 IHE profiles have been identified for referencing in public procurement, such as XDS, XDS-I, PIX, and PDQ.<sup id=\"rdp-ebb-cite_ref-ECCommiss15_42-0\" class=\"reference\"><a href=\"#cite_note-ECCommiss15-42\">[42]<\/a><\/sup>\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">Cloud computing<\/a> is now used everywhere and provides an important set of features, such as, adaptive scalability, performance, and benefits from the business perspective. In particular, the sharing of clinical information in the cloud can be very advantageous, especially in cases where central repositories or central coordination are needed. But organizations should also be wary about the data protection, privacy, and access control mechanisms that should be in place<sup id=\"rdp-ebb-cite_ref-SpanakisSecure16_43-0\" class=\"reference\"><a href=\"#cite_note-SpanakisSecure16-43\">[43]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrammatikisSecuring19_44-0\" class=\"reference\"><a href=\"#cite_note-GrammatikisSecuring19-44\">[44]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PasqualeProtect14_45-0\" class=\"reference\"><a href=\"#cite_note-PasqualeProtect14-45\">[45]<\/a><\/sup>, either offered by the cloud provider or built in-house, in order to properly handle sensitive data and comply with regulations such as GDPR.\n<\/p><p>Blockchain is a highly interesting technology that can be put to good use in information sharing, more specifically for supporting decentralization, data integrity verification, and data auditability. These inherent features of blockchain have been praised and discussed in the context of healthcare as valuable tools.<sup id=\"rdp-ebb-cite_ref-AngraalBlock17_41-1\" class=\"reference\"><a href=\"#cite_note-AngraalBlock17-41\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LinnBlock16_46-0\" class=\"reference\"><a href=\"#cite_note-LinnBlock16-46\">[46]<\/a><\/sup> Nevertheless, there are some major issues to be resolved, such as the compliance with GDPR's \u201cright to be forgotten\u201d requirement, which, unless the blockchain implementation is adapted, requires the deletion of data from the blockchain when the data owner asks to do it, which is not feasible, by design, in the \u201ctraditional\u201d blockchain implementations. Furthermore, there is also the additional complexity followed by the huge heterogeneity of medical data (i.e., text and images) that do not fit exactly to the original design of blockchain. It is evident that such requirements imposed by GDPR and the application domain present challenges and introduce additional trade-offs related to the management of data, administration, and overall governance. For example, storing the health data \u201coff-chain\u201d (i.e., external to the blockchain network) and only metadata \u201con-chain\u201d may introduce problems of availability, performance, data protection, and integrity.<sup id=\"rdp-ebb-cite_ref-MiyachiHOCBS21_47-0\" class=\"reference\"><a href=\"#cite_note-MiyachiHOCBS21-47\">[47]<\/a><\/sup> Therefore, careful considerations of the available options should be made before committing to such cutting-edge technologies.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>ES, SS, SB, and CC made core contribution to the design of the study, collection of information, guidelines, established practices, and standards and made an extensive evaluation on new and promising technologies for the implementation of a secure information sharing platform for health-related data including blockchain evaluation. SM and VS coordinated the work in respect of the project PANACEA. All authors contributed to the article and approved the submitted version.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_availability_statement\">Data availability statement<\/span><\/h3>\n<p>The original contributions generated for the study are included in the article\/supplementary material; further inquiries can be directed to the corresponding author\/s.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work has been supported by the PANACEA project, which has received funding from the European Union's Horizon 2020 research and innovation program under the Grant Agreement No. 826293.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-CongressBringing95-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CongressBringing95_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">U.S. Congress, Office of Technology Assessment (September 1995). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ota.fas.org\/reports\/9507.pdf\" target=\"_blank\">\"Bringing Health Care Online: The Role of Information Technologies\"<\/a> (PDF). U.S. Government Printing Office<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ota.fas.org\/reports\/9507.pdf\" target=\"_blank\">https:\/\/ota.fas.org\/reports\/9507.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bringing+Health+Care+Online%3A+The+Role+of+Information+Technologies&rft.atitle=&rft.aulast=U.S.+Congress%2C+Office+of+Technology+Assessment&rft.au=U.S.+Congress%2C+Office+of+Technology+Assessment&rft.date=September+1995&rft.pub=U.S.+Government+Printing+Office&rft_id=https%3A%2F%2Fota.fas.org%2Freports%2F9507.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KolodnerHealth08-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KolodnerHealth08_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kolodner, R.M.; Cohn, S.P.; Freidman, C.P. (2008). \"Health Information Technology: Strategic Initiatives, Real Progress\". <i>Health Affairs<\/i> <b>27<\/b> (Supp. 1): w391\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.27.5.w391\" target=\"_blank\">10.1377\/hlthaff.27.5.w391<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Health+Information+Technology%3A+Strategic+Initiatives%2C+Real+Progress&rft.jtitle=Health+Affairs&rft.aulast=Kolodner%2C+R.M.%3B+Cohn%2C+S.P.%3B+Freidman%2C+C.P.&rft.au=Kolodner%2C+R.M.%3B+Cohn%2C+S.P.%3B+Freidman%2C+C.P.&rft.date=2008&rft.volume=27&rft.issue=Supp.+1&rft.pages=w391%E2%80%935&rft_id=info:doi\/10.1377%2Fhlthaff.27.5.w391&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SpanakisCyber20-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SpanakisCyber20_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Spanakis, E.G.; Bonomi, S.; Sfakianakis, S. (2020). \"Cyber-attacks and threats for healthcare - A multi-layer thread analysis\". <i>Proceedings of the 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society<\/i>: 5705\u201308. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FEMBC44109.2020.9176698\" target=\"_blank\">10.1109\/EMBC44109.2020.9176698<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/33019270\" target=\"_blank\">33019270<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyber-attacks+and+threats+for+healthcare+-+A+multi-layer+thread+analysis&rft.jtitle=Proceedings+of+the+42nd+Annual+International+Conference+of+the+IEEE+Engineering+in+Medicine+%26+Biology+Society&rft.aulast=Spanakis%2C+E.G.%3B+Bonomi%2C+S.%3B+Sfakianakis%2C+S.&rft.au=Spanakis%2C+E.G.%3B+Bonomi%2C+S.%3B+Sfakianakis%2C+S.&rft.date=2020&rft.pages=5705%E2%80%9308&rft_id=info:doi\/10.1109%2FEMBC44109.2020.9176698&rft_id=info:pmid\/33019270&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VanVelsenAMaturity16-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VanVelsenAMaturity16_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-VanVelsenAMaturity16_4-1\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Van Velsen, L.; Hermens, H.; d'Hollosy, O.-N. (2016). \"A maturity model for interoperability in eHealth\". <i>Proceedings of the IEEE 18th International Conference on e-Health Networking, Applications and Services<\/i>: 1\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FHealthCom.2016.7749533\" target=\"_blank\">10.1109\/HealthCom.2016.7749533<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+maturity+model+for+interoperability+in+eHealth&rft.jtitle=Proceedings+of+the+IEEE+18th+International+Conference+on+e-Health+Networking%2C+Applications+and+Services&rft.aulast=Van+Velsen%2C+L.%3B+Hermens%2C+H.%3B+d%27Hollosy%2C+O.-N.&rft.au=Van+Velsen%2C+L.%3B+Hermens%2C+H.%3B+d%27Hollosy%2C+O.-N.&rft.date=2016&rft.pages=1%E2%80%936&rft_id=info:doi\/10.1109%2FHealthCom.2016.7749533&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChronakiAneHealth03-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChronakiAneHealth03_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chronaki, C.E.; Chiarugi, F.; Mavrogiannaki, E. et al. (2003). \"An eHealth platform for instant interaction among health professionals\". <i>Proceedings of the Computers in Cardiology 2003<\/i>: 101\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCIC.2003.1291100\" target=\"_blank\">10.1109\/CIC.2003.1291100<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+eHealth+platform+for+instant+interaction+among+health+professionals&rft.jtitle=Proceedings+of+the+Computers+in+Cardiology+2003&rft.aulast=Chronaki%2C+C.E.%3B+Chiarugi%2C+F.%3B+Mavrogiannaki%2C+E.+et+al.&rft.au=Chronaki%2C+C.E.%3B+Chiarugi%2C+F.%3B+Mavrogiannaki%2C+E.+et+al.&rft.date=2003&rft.pages=101%E2%80%934&rft_id=info:doi\/10.1109%2FCIC.2003.1291100&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SpanakisR.26D05-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SpanakisR.26D05_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Spanakis, M; Lelis, P.; Chiarugi, F. et al. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/139.91.210.27\/CBML\/PROCEEDINGS\/2005_EMBEC\/Embec%202005\/Abstracts\/Abstract791.html\" target=\"_blank\">\"R&D Challenges in Developing an Ambient Intelligence eHealth Platform\"<\/a>. <i>IFMBE Proceedings<\/i> <b>11<\/b> (1): 1727\u2013983<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/139.91.210.27\/CBML\/PROCEEDINGS\/2005_EMBEC\/Embec%202005\/Abstracts\/Abstract791.html\" target=\"_blank\">http:\/\/139.91.210.27\/CBML\/PROCEEDINGS\/2005_EMBEC\/Embec%202005\/Abstracts\/Abstract791.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=R%26D+Challenges+in+Developing+an+Ambient+Intelligence+eHealth+Platform&rft.jtitle=IFMBE+Proceedings&rft.aulast=Spanakis%2C+M%3B+Lelis%2C+P.%3B+Chiarugi%2C+F.+et+al.&rft.au=Spanakis%2C+M%3B+Lelis%2C+P.%3B+Chiarugi%2C+F.+et+al.&rft.date=2005&rft.volume=11&rft.issue=1&rft.pages=1727%E2%80%93983&rft_id=http%3A%2F%2F139.91.210.27%2FCBML%2FPROCEEDINGS%2F2005_EMBEC%2FEmbec%25202005%2FAbstracts%2FAbstract791.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TsiknakisAdopt10-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TsiknakisAdopt10_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tsiknakis, M.; Spanakis, M. (2010). \"Adoption of innovative eHealth services in prehospital emergency management: A case study\". <i>Proceedings of the 10th IEEE International Conference on Information Technology and Applications in Biomedicine<\/i>: 1\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FITAB.2010.5687752\" target=\"_blank\">10.1109\/ITAB.2010.5687752<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adoption+of+innovative+eHealth+services+in+prehospital+emergency+management%3A+A+case+study&rft.jtitle=Proceedings+of+the+10th+IEEE+International+Conference+on+Information+Technology+and+Applications+in+Biomedicine&rft.aulast=Tsiknakis%2C+M.%3B+Spanakis%2C+M.&rft.au=Tsiknakis%2C+M.%3B+Spanakis%2C+M.&rft.date=2010&rft.pages=1%E2%80%935&rft_id=info:doi\/10.1109%2FITAB.2010.5687752&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SpanakisCongest18-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SpanakisCongest18_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Spanakis, E.G.; Psaraki, M.; Sakkalis, V. (2018). \"Congestive Heart Failure Risk Assessment Monitoring through Internet of things and mobile Personal Health Systems\". <i>Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society<\/i>: 2925-28. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FEMBC.2018.8513024\" target=\"_blank\">10.1109\/EMBC.2018.8513024<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30441013\" target=\"_blank\">30441013<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Congestive+Heart+Failure+Risk+Assessment+Monitoring+through+Internet+of+things+and+mobile+Personal+Health+Systems&rft.jtitle=Proceedings+of+the+40th+Annual+International+Conference+of+the+IEEE+Engineering+in+Medicine+and+Biology+Society&rft.aulast=Spanakis%2C+E.G.%3B+Psaraki%2C+M.%3B+Sakkalis%2C+V.&rft.au=Spanakis%2C+E.G.%3B+Psaraki%2C+M.%3B+Sakkalis%2C+V.&rft.date=2018&rft.pages=2925-28&rft_id=info:doi\/10.1109%2FEMBC.2018.8513024&rft_id=info:pmid\/30441013&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SpanakisPharm19-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SpanakisPharm19_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Spanakis, M.; Sfakianakas, S.; Sakkalis, V. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6473432\" target=\"_blank\">\"PharmActa: Empowering Patients to Avoid Clinical Significant Drug\u207bHerb Interactions\"<\/a>. <i>Medicines<\/i> <b>6<\/b> (1): 26. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fmedicines6010026\" target=\"_blank\">10.3390\/medicines6010026<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6473432\/\" target=\"_blank\">PMC6473432<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30781500\" target=\"_blank\">30781500<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6473432\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6473432<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PharmActa%3A+Empowering+Patients+to+Avoid+Clinical+Significant+Drug%E2%81%BBHerb+Interactions&rft.jtitle=Medicines&rft.aulast=Spanakis%2C+M.%3B+Sfakianakas%2C+S.%3B+Sakkalis%2C+V.+et+al.&rft.au=Spanakis%2C+M.%3B+Sfakianakas%2C+S.%3B+Sakkalis%2C+V.+et+al.&rft.date=2019&rft.volume=6&rft.issue=1&rft.pages=26&rft_id=info:doi\/10.3390%2Fmedicines6010026&rft_id=info:pmc\/PMC6473432&rft_id=info:pmid\/30781500&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6473432&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EUeHealth15-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EUeHealth15_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ec.europa.eu\/health\/sites\/default\/files\/ehealth\/docs\/ev_20151123_co03_en.pdf\" target=\"_blank\">\"eHealth Network: Refined eHealth European Interoperability Framework\"<\/a> (PDF). European Union. 23 November 2015. pp. 24<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ec.europa.eu\/health\/sites\/default\/files\/ehealth\/docs\/ev_20151123_co03_en.pdf\" target=\"_blank\">https:\/\/ec.europa.eu\/health\/sites\/default\/files\/ehealth\/docs\/ev_20151123_co03_en.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=eHealth+Network%3A+Refined+eHealth+European+Interoperability+Framework&rft.atitle=&rft.date=23+November+2015&rft.pages=pp.+24&rft.pub=European+Union&rft_id=https%3A%2F%2Fec.europa.eu%2Fhealth%2Fsites%2Fdefault%2Ffiles%2Fehealth%2Fdocs%2Fev_20151123_co03_en.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KatehakisSecurity17-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KatehakisSecurity17_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Katehakis, D.G.; Pangalos, G.; Prentza, A. (2017). \"Security Improvements for Better and Safer Cross-Border ePrescription and Patient Summary Services\". <i>International Journal of Reliable and Quality E-Healthcare<\/i> <b>6<\/b> (1): 2. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4018%2FIJRQEH.2017010102\" target=\"_blank\">10.4018\/IJRQEH.2017010102<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Security+Improvements+for+Better+and+Safer+Cross-Border+ePrescription+and+Patient+Summary+Services&rft.jtitle=International+Journal+of+Reliable+and+Quality+E-Healthcare&rft.aulast=Katehakis%2C+D.G.%3B+Pangalos%2C+G.%3B+Prentza%2C+A.&rft.au=Katehakis%2C+D.G.%3B+Pangalos%2C+G.%3B+Prentza%2C+A.&rft.date=2017&rft.volume=6&rft.issue=1&rft.pages=2&rft_id=info:doi\/10.4018%2FIJRQEH.2017010102&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TinholtStudy13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TinholtStudy13_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Tinholt, D.; Carrara, W.; Tol, T. et al. (18 June 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/final-report-study-analysis-needs-cross-border-services-and-assessment-organisational-legal\" target=\"_blank\">\"Final Report - Study on Analysis of the Needs for Cross-Border Services and Assessment of the Organisational, Legal, Technical and Semantic Barriers (SMART 2011\/0074)\"<\/a>. European Commission. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2759%2F10003\" target=\"_blank\">10.2759\/10003<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/final-report-study-analysis-needs-cross-border-services-and-assessment-organisational-legal\" target=\"_blank\">https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/final-report-study-analysis-needs-cross-border-services-and-assessment-organisational-legal<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Final+Report+-+Study+on+Analysis+of+the+Needs+for+Cross-Border+Services+and+Assessment+of+the+Organisational%2C+Legal%2C+Technical+and+Semantic+Barriers+%28SMART+2011%2F0074%29&rft.atitle=&rft.aulast=Tinholt%2C+D.%3B+Carrara%2C+W.%3B+Tol%2C+T.+et+al.&rft.au=Tinholt%2C+D.%3B+Carrara%2C+W.%3B+Tol%2C+T.+et+al.&rft.date=18+June+2013&rft.pub=European+Commission&rft_id=info:doi\/10.2759%2F10003&rft_id=https%3A%2F%2Fec.europa.eu%2Fdigital-single-market%2Fen%2Fnews%2Ffinal-report-study-analysis-needs-cross-border-services-and-assessment-organisational-legal&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PeetersFree12-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PeetersFree12_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Peeters, M. (2012). \"Free movement of patients: Directive 2011\/24 on the application of patients' rights in cross-border healthcare\". <i>European Journal of Health Law<\/i> <b>19<\/b> (1): 29\u201360. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1163%2F157180912x615158\" target=\"_blank\">10.1163\/157180912x615158<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22428388\" target=\"_blank\">22428388<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Free+movement+of+patients%3A+Directive+2011%2F24+on+the+application+of+patients%27+rights+in+cross-border+healthcare&rft.jtitle=European+Journal+of+Health+Law&rft.aulast=Peeters%2C+M.&rft.au=Peeters%2C+M.&rft.date=2012&rft.volume=19&rft.issue=1&rft.pages=29%E2%80%9360&rft_id=info:doi\/10.1163%2F157180912x615158&rft_id=info:pmid\/22428388&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BensonPrinc12-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BensonPrinc12_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Benson, T. (2012). <i>Principles of Health Interoperability HL7 and SNOMED<\/i> (2nd ed.). Springer. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-1-4471-2801-4\" target=\"_blank\">10.1007\/978-1-4471-2801-4<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781447128014.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Principles+of+Health+Interoperability+HL7+and+SNOMED&rft.aulast=Benson%2C+T.&rft.au=Benson%2C+T.&rft.date=2012&rft.edition=2nd&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-1-4471-2801-4&rft.isbn=9781447128014&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KuchinkeCDISC09-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KuchinkeCDISC09_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kuchinke, W.; Alerts, J.; Semler, S.C. et al. (2009). \"CDISC standard-based electronic archiving of clinical trials\". <i>Methods of Information in Medicine<\/i> <b>48<\/b> (5): 408\u201313. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3414%2FME9236\" target=\"_blank\">10.3414\/ME9236<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19621114\" target=\"_blank\">19621114<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=CDISC+standard-based+electronic+archiving+of+clinical+trials&rft.jtitle=Methods+of+Information+in+Medicine&rft.aulast=Kuchinke%2C+W.%3B+Alerts%2C+J.%3B+Semler%2C+S.C.+et+al.&rft.au=Kuchinke%2C+W.%3B+Alerts%2C+J.%3B+Semler%2C+S.C.+et+al.&rft.date=2009&rft.volume=48&rft.issue=5&rft.pages=408%E2%80%9313&rft_id=info:doi\/10.3414%2FME9236&rft_id=info:pmid\/19621114&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarrollContinua07-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CarrollContinua07_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Carroll, R.; Cnossen, R.; Schnell, M. et al. (2007). \"Continua: An Interoperable Personal Healthcare Ecosystem\". <i>IEEE Pervasive Computing<\/i> <b>6<\/b> (4): 90\u201394. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FMPRV.2007.72\" target=\"_blank\">10.1109\/MPRV.2007.72<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Continua%3A+An+Interoperable+Personal+Healthcare+Ecosystem&rft.jtitle=IEEE+Pervasive+Computing&rft.aulast=Carroll%2C+R.%3B+Cnossen%2C+R.%3B+Schnell%2C+M.+et+al.&rft.au=Carroll%2C+R.%3B+Cnossen%2C+R.%3B+Schnell%2C+M.+et+al.&rft.date=2007&rft.volume=6&rft.issue=4&rft.pages=90%E2%80%9394&rft_id=info:doi\/10.1109%2FMPRV.2007.72&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MariasClin09-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MariasClin09_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Marias, K.; Sakkalis, A.; Roniotis, C. et al. (2009). \"Clinically Oriented Translational Cancer Multilevel Modeling: The ContraCancrum Project\". <i>Proceedings of the 2009 World Congress on Medical Physics and Biomedical Engineering<\/i>: 2121\u201327. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-03882-2_564\" target=\"_blank\">10.1007\/978-3-642-03882-2_564<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Clinically+Oriented+Translational+Cancer+Multilevel+Modeling%3A+The+ContraCancrum+Project&rft.jtitle=Proceedings+of+the+2009+World+Congress+on+Medical+Physics+and+Biomedical+Engineering&rft.aulast=Marias%2C+K.%3B+Sakkalis%2C+A.%3B+Roniotis%2C+C.+et+al.&rft.au=Marias%2C+K.%3B+Sakkalis%2C+A.%3B+Roniotis%2C+C.+et+al.&rft.date=2009&rft.pages=2121%E2%80%9327&rft_id=info:doi\/10.1007%2F978-3-642-03882-2_564&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LarobinaMedical13-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LarobinaMedical13_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Larobina, M.; Murino, L. (2014). \"Medical Image File Formats\". <i>Journal of Digital Imaging<\/i> <b>27<\/b>: 200\u201306. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-013-9657-9\" target=\"_blank\">10.1007\/s10278-013-9657-9<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+Image+File+Formats&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=Larobina%2C+M.%3B+Murino%2C+L.&rft.au=Larobina%2C+M.%3B+Murino%2C+L.&rft.date=2014&rft.volume=27&rft.pages=200%E2%80%9306&rft_id=info:doi\/10.1007%2Fs10278-013-9657-9&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CockTheSanger10-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CockTheSanger10_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cock, P.J.A.; Fields, C.J.; Goto, N. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2847217\" target=\"_blank\">\"The Sanger FASTQ file format for sequences with quality scores, and the Solexa\/Illumina FASTQ variants\"<\/a>. <i>Nucleic Acids Research<\/i> <b>38<\/b> (6): 1767\u201371. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkp1137\" target=\"_blank\">10.1093\/nar\/gkp1137<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2847217\/\" target=\"_blank\">PMC2847217<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20015970\" target=\"_blank\">20015970<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2847217\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2847217<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Sanger+FASTQ+file+format+for+sequences+with+quality+scores%2C+and+the+Solexa%2FIllumina+FASTQ+variants&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Cock%2C+P.J.A.%3B+Fields%2C+C.J.%3B+Goto%2C+N.+et+al.&rft.au=Cock%2C+P.J.A.%3B+Fields%2C+C.J.%3B+Goto%2C+N.+et+al.&rft.date=2010&rft.volume=38&rft.issue=6&rft.pages=1767%E2%80%9371&rft_id=info:doi\/10.1093%2Fnar%2Fgkp1137&rft_id=info:pmc\/PMC2847217&rft_id=info:pmid\/20015970&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2847217&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeinonenTheSeq11-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeinonenTheSeq11_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leinonen, R.; Sugawara, H.; Shumway, M. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3013647\" target=\"_blank\">\"The sequence read archive\"<\/a>. <i>Nucleic Acids Research<\/i> <b>39<\/b> (DB1): D19\u201321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkq1019\" target=\"_blank\">10.1093\/nar\/gkq1019<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3013647\/\" target=\"_blank\">PMC3013647<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21062823\" target=\"_blank\">21062823<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3013647\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3013647<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+sequence+read+archive&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Leinonen%2C+R.%3B+Sugawara%2C+H.%3B+Shumway%2C+M.+et+al.&rft.au=Leinonen%2C+R.%3B+Sugawara%2C+H.%3B+Shumway%2C+M.+et+al.&rft.date=2011&rft.volume=39&rft.issue=DB1&rft.pages=D19%E2%80%9321&rft_id=info:doi\/10.1093%2Fnar%2Fgkq1019&rft_id=info:pmc\/PMC3013647&rft_id=info:pmid\/21062823&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3013647&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DolinTheHL7_01-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DolinTheHL7_01_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dolin, R.H.; Alschuler, L.; Beebe, C. et al. (2001). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC130066\" target=\"_blank\">\"The HL7 Clinical Document Architecture\"<\/a>. <i>JAMIA<\/i> <b>8<\/b> (6): 552\u201369. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fjamia.2001.0080552\" target=\"_blank\">10.1136\/jamia.2001.0080552<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC130066\/\" target=\"_blank\">PMC130066<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11687563\" target=\"_blank\">11687563<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC130066\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC130066<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+HL7+Clinical+Document+Architecture&rft.jtitle=JAMIA&rft.aulast=Dolin%2C+R.H.%3B+Alschuler%2C+L.%3B+Beebe%2C+C.+et+al.&rft.au=Dolin%2C+R.H.%3B+Alschuler%2C+L.%3B+Beebe%2C+C.+et+al.&rft.date=2001&rft.volume=8&rft.issue=6&rft.pages=552%E2%80%9369&rft_id=info:doi\/10.1136%2Fjamia.2001.0080552&rft_id=info:pmc\/PMC130066&rft_id=info:pmid\/11687563&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC130066&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Freitas-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Freitas_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Freitas, F.; Schulz, S.; Moraes, E. (2009). \"Survey of current terminologies and ontologies in biology and medicine\". <i>RECIIS<\/i> <b>3<\/b> (1): 239\u201349. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3395%2Freciis.v3i1.239en\" target=\"_blank\">10.3395\/reciis.v3i1.239en<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Survey+of+current+terminologies+and+ontologies+in+biology+and+medicine&rft.jtitle=RECIIS&rft.aulast=Freitas%2C+F.%3B+Schulz%2C+S.%3B+Moraes%2C+E.&rft.au=Freitas%2C+F.%3B+Schulz%2C+S.%3B+Moraes%2C+E.&rft.date=2009&rft.volume=3&rft.issue=1&rft.pages=239%E2%80%9349&rft_id=info:doi\/10.3395%2Freciis.v3i1.239en&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BenderHL713-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BenderHL713_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bender, D.; Sartipi, K. (2013). \"HL7 FHIR: An Agile and RESTful approach to healthcare information exchange\". <i>Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems<\/i>: 326\u201331. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCBMS.2013.6627810\" target=\"_blank\">10.1109\/CBMS.2013.6627810<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=HL7+FHIR%3A+An+Agile+and+RESTful+approach+to+healthcare+information+exchange&rft.jtitle=Proceedings+of+the+26th+IEEE+International+Symposium+on+Computer-Based+Medical+Systems&rft.aulast=Bender%2C+D.%3B+Sartipi%2C+K.&rft.au=Bender%2C+D.%3B+Sartipi%2C+K.&rft.date=2013&rft.pages=326%E2%80%9331&rft_id=info:doi\/10.1109%2FCBMS.2013.6627810&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KondylakisUsing19-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KondylakisUsing19_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kondylakis, H.; Petrakis, Y.; Leivadoros, S. et al. (2019). \"Using XDS and FHIR to Support Mobile Access to EHR Information Through Personal Health Apps\". <i>Proceedings of the 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems<\/i>: 241\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCBMS.2019.00058\" target=\"_blank\">10.1109\/CBMS.2019.00058<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+XDS+and+FHIR+to+Support+Mobile+Access+to+EHR+Information+Through+Personal+Health+Apps&rft.jtitle=Proceedings+of+the+2019+IEEE+32nd+International+Symposium+on+Computer-Based+Medical+Systems&rft.aulast=Kondylakis%2C+H.%3B+Petrakis%2C+Y.%3B+Leivadoros%2C+S.+et+al.&rft.au=Kondylakis%2C+H.%3B+Petrakis%2C+Y.%3B+Leivadoros%2C+S.+et+al.&rft.date=2019&rft.pages=241%E2%80%934&rft_id=info:doi\/10.1109%2FCBMS.2019.00058&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EckmanVarieties07-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EckmanVarieties07_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Eckman, B.A.; Bennett, C.A.; Kaufman, J.H. et al. (2007). \"Varieties of interoperability in the transformation of the health-care information infrastructure\". <i>IBM Systems Journal<\/i> <b>46<\/b> (1): 19\u201341. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1147%2Fsj.461.0019\" target=\"_blank\">10.1147\/sj.461.0019<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Varieties+of+interoperability+in+the+transformation+of+the+health-care+information+infrastructure&rft.jtitle=IBM+Systems+Journal&rft.aulast=Eckman%2C+B.A.%3B+Bennett%2C+C.A.%3B+Kaufman%2C+J.H.+et+al.&rft.au=Eckman%2C+B.A.%3B+Bennett%2C+C.A.%3B+Kaufman%2C+J.H.+et+al.&rft.date=2007&rft.volume=46&rft.issue=1&rft.pages=19%E2%80%9341&rft_id=info:doi\/10.1147%2Fsj.461.0019&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NikoloudakisVulner19-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NikoloudakisVulner19_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nikoloudakis, Y.; Pallis, E.; Mastorakis, G. et al. (2019). \"Vulnerability assessment as a service for fog-centric ICT ecosystems: A healthcare use case\". <i>Peer-to-Peer Networking and Applications<\/i> <b>12<\/b>: 1216\u201324. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs12083-019-0716-y\" target=\"_blank\">10.1007\/s12083-019-0716-y<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Vulnerability+assessment+as+a+service+for+fog-centric+ICT+ecosystems%3A+A+healthcare+use+case&rft.jtitle=Peer-to-Peer+Networking+and+Applications&rft.aulast=Nikoloudakis%2C+Y.%3B+Pallis%2C+E.%3B+Mastorakis%2C+G.+et+al.&rft.au=Nikoloudakis%2C+Y.%3B+Pallis%2C+E.%3B+Mastorakis%2C+G.+et+al.&rft.date=2019&rft.volume=12&rft.pages=1216%E2%80%9324&rft_id=info:doi\/10.1007%2Fs12083-019-0716-y&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NoferBlock17-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NoferBlock17_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nofer, M.; Gomber, P.; Hinz, O. (2017). \"Blockchain\". <i>Business & Information Systems Engineering<\/i> <b>59<\/b>: 183\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs12599-017-0467-3\" target=\"_blank\">10.1007\/s12599-017-0467-3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Blockchain&rft.jtitle=Business+%26+Information+Systems+Engineering&rft.aulast=Nofer%2C+M.%3B+Gomber%2C+P.%3B+Hinz%2C+O.&rft.au=Nofer%2C+M.%3B+Gomber%2C+P.%3B+Hinz%2C+O.&rft.date=2017&rft.volume=59&rft.pages=183%E2%80%937&rft_id=info:doi\/10.1007%2Fs12599-017-0467-3&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VergneDecentral20-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VergneDecentral20_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vergne, J.P. (2020). \"Decentralized vs. Distributed Organization: Blockchain, Machine Learning and the Future of the Digital Platform\". <i>Organization Theory<\/i> <b>1<\/b> (4). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F2631787720977052\" target=\"_blank\">10.1177\/2631787720977052<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Decentralized+vs.+Distributed+Organization%3A+Blockchain%2C+Machine+Learning+and+the+Future+of+the+Digital+Platform&rft.jtitle=Organization+Theory&rft.aulast=Vergne%2C+J.P.&rft.au=Vergne%2C+J.P.&rft.date=2020&rft.volume=1&rft.issue=4&rft_id=info:doi\/10.1177%2F2631787720977052&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MacdonaldTheBlock17-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MacdonaldTheBlock17_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Macdonald, M.; Liu-Thorrold, L.; Julien, R. (February 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.researchgate.net\/publication\/313249614_The_Blockchain_A_Comparison_of_Platforms_and_Their_Uses_Beyond_Bitcoin\" target=\"_blank\">\"The Blockchain: A Comparison of Platforms and Their Uses Beyond Bitcoin\"<\/a>. <i>ResearchGate<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.13140%2FRG.2.2.23274.52164\" target=\"_blank\">10.13140\/RG.2.2.23274.52164<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.researchgate.net\/publication\/313249614_The_Blockchain_A_Comparison_of_Platforms_and_Their_Uses_Beyond_Bitcoin\" target=\"_blank\">https:\/\/www.researchgate.net\/publication\/313249614_The_Blockchain_A_Comparison_of_Platforms_and_Their_Uses_Beyond_Bitcoin<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Blockchain%3A+A+Comparison+of+Platforms+and+Their+Uses+Beyond+Bitcoin&rft.atitle=ResearchGate&rft.aulast=Macdonald%2C+M.%3B+Liu-Thorrold%2C+L.%3B+Julien%2C+R.&rft.au=Macdonald%2C+M.%3B+Liu-Thorrold%2C+L.%3B+Julien%2C+R.&rft.date=February+2017&rft_id=info:doi\/10.13140%2FRG.2.2.23274.52164&rft_id=https%3A%2F%2Fwww.researchgate.net%2Fpublication%2F313249614_The_Blockchain_A_Comparison_of_Platforms_and_Their_Uses_Beyond_Bitcoin&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KuoComp19-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KuoComp19_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kuo, T.-T.; Rojas, H.Z.; Ohno-Machado, L. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7787359\" target=\"_blank\">\"Comparison of blockchain platforms: A systematic review and healthcare examples\"<\/a>. <i>JAMIA<\/i> <b>26<\/b> (5): 462\u201378. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fjamia%2Focy185\" target=\"_blank\">10.1093\/jamia\/ocy185<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7787359\/\" target=\"_blank\">PMC7787359<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30907419\" target=\"_blank\">30907419<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7787359\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7787359<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comparison+of+blockchain+platforms%3A+A+systematic+review+and+healthcare+examples&rft.jtitle=JAMIA&rft.aulast=Kuo%2C+T.-T.%3B+Rojas%2C+H.Z.%3B+Ohno-Machado%2C+L.&rft.au=Kuo%2C+T.-T.%3B+Rojas%2C+H.Z.%3B+Ohno-Machado%2C+L.&rft.date=2019&rft.volume=26&rft.issue=5&rft.pages=462%E2%80%9378&rft_id=info:doi\/10.1093%2Fjamia%2Focy185&rft_id=info:pmc\/PMC7787359&rft_id=info:pmid\/30907419&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7787359&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChowdhuryAComp19-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChowdhuryAComp19_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chowdhury, M.J.M.; Ferdous, S.; Biswas, K. et al. (2019). \"A Comparative Analysis of Distributed Ledger Technology Platforms\". <i>IEEE Access<\/i> <b>7<\/b>: 167930\u201343. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2953729\" target=\"_blank\">10.1109\/ACCESS.2019.2953729<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Comparative+Analysis+of+Distributed+Ledger+Technology+Platforms&rft.jtitle=IEEE+Access&rft.aulast=Chowdhury%2C+M.J.M.%3B+Ferdous%2C+S.%3B+Biswas%2C+K.+et+al.&rft.au=Chowdhury%2C+M.J.M.%3B+Ferdous%2C+S.%3B+Biswas%2C+K.+et+al.&rft.date=2019&rft.volume=7&rft.pages=167930%E2%80%9343&rft_id=info:doi\/10.1109%2FACCESS.2019.2953729&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TriantafyllouTheChall20-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TriantafyllouTheChall20_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Triantafyllou, A.; Jimenez, J.A.P.; Torres, A.D.R. et al. (2020). \"The Challenges of Privacy and Access Control as Key Perspectives for the Future Electric Smart Grid\". <i>IEEE Open Journal of the Communications Society<\/i> <b>1<\/b>: 1934\u201360. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FOJCOMS.2020.3037517\" target=\"_blank\">10.1109\/OJCOMS.2020.3037517<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Challenges+of+Privacy+and+Access+Control+as+Key+Perspectives+for+the+Future+Electric+Smart+Grid&rft.jtitle=IEEE+Open+Journal+of+the+Communications+Society&rft.aulast=Triantafyllou%2C+A.%3B+Jimenez%2C+J.A.P.%3B+Torres%2C+A.D.R.+et+al.&rft.au=Triantafyllou%2C+A.%3B+Jimenez%2C+J.A.P.%3B+Torres%2C+A.D.R.+et+al.&rft.date=2020&rft.volume=1&rft.pages=1934%E2%80%9360&rft_id=info:doi\/10.1109%2FOJCOMS.2020.3037517&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GrammatikisARIES20-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GrammatikisARIES20_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Grammatikis, P.R.; Sarigiannidis, P.; Efstathopoulos, G. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7570496\" target=\"_blank\">\"ARIES: A Novel Multivariate Intrusion Detection System for Smart Grid\"<\/a>. <i>Sensors<\/i> <b>20<\/b> (18): 5305. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs20185305\" target=\"_blank\">10.3390\/s20185305<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7570496\/\" target=\"_blank\">PMC7570496<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32948064\" target=\"_blank\">32948064<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7570496\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7570496<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ARIES%3A+A+Novel+Multivariate+Intrusion+Detection+System+for+Smart+Grid&rft.jtitle=Sensors&rft.aulast=Grammatikis%2C+P.R.%3B+Sarigiannidis%2C+P.%3B+Efstathopoulos%2C+G.+et+al.&rft.au=Grammatikis%2C+P.R.%3B+Sarigiannidis%2C+P.%3B+Efstathopoulos%2C+G.+et+al.&rft.date=2020&rft.volume=20&rft.issue=18&rft.pages=5305&rft_id=info:doi\/10.3390%2Fs20185305&rft_id=info:pmc\/PMC7570496&rft_id=info:pmid\/32948064&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7570496&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChocklerGroup01-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChocklerGroup01_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chockler, G.V.; Keidar, I.; Vitenberg, R. (2001). \"Group communication specifications: A comprehensive study\". <i>ACM Computing Surveys<\/i> <b>33<\/b> (4): 427\u201369. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F503112.503113\" target=\"_blank\">10.1145\/503112.503113<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Group+communication+specifications%3A+A+comprehensive+study&rft.jtitle=ACM+Computing+Surveys&rft.aulast=Chockler%2C+G.V.%3B+Keidar%2C+I.%3B+Vitenberg%2C+R.&rft.au=Chockler%2C+G.V.%3B+Keidar%2C+I.%3B+Vitenberg%2C+R.&rft.date=2001&rft.volume=33&rft.issue=4&rft.pages=427%E2%80%9369&rft_id=info:doi\/10.1145%2F503112.503113&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AguileraDynamic11-35\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AguileraDynamic11_35-0\">35.0<\/a><\/sup> <sup><a href=\"#cite_ref-AguileraDynamic11_35-1\">35.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aguilera, M.K.; Keidar, I.; Malkhi, D. et al. (2011). \"Dynamic atomic storage without consensus\". <i>Journal of the ACM<\/i> <b>58<\/b> (2): 1\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1944345.1944348\" target=\"_blank\">10.1145\/1944345.1944348<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Dynamic+atomic+storage+without+consensus&rft.jtitle=Journal+of+the+ACM&rft.aulast=Aguilera%2C+M.K.%3B+Keidar%2C+I.%3B+Malkhi%2C+D.+et+al.&rft.au=Aguilera%2C+M.K.%3B+Keidar%2C+I.%3B+Malkhi%2C+D.+et+al.&rft.date=2011&rft.volume=58&rft.issue=2&rft.pages=1%E2%80%9332&rft_id=info:doi\/10.1145%2F1944345.1944348&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoieraEConsent04-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CoieraEConsent04_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coiera, E.; Clarke, R. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC353020\" target=\"_blank\">\"e-Consent: The design and implementation of consumer consent mechanisms in an electronic environment\"<\/a>. <i>JAMIA<\/i> <b>11<\/b> (2): 129\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M1480\" target=\"_blank\">10.1197\/jamia.M1480<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC353020\/\" target=\"_blank\">PMC353020<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/14662803\" target=\"_blank\">14662803<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC353020\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC353020<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=e-Consent%3A+The+design+and+implementation+of+consumer+consent+mechanisms+in+an+electronic+environment&rft.jtitle=JAMIA&rft.aulast=Coiera%2C+E.%3B+Clarke%2C+R.&rft.au=Coiera%2C+E.%3B+Clarke%2C+R.&rft.date=2004&rft.volume=11&rft.issue=2&rft.pages=129%E2%80%9340&rft_id=info:doi\/10.1197%2Fjamia.M1480&rft_id=info:pmc\/PMC353020&rft_id=info:pmid\/14662803&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC353020&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BergmannEneConsent05-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BergmannEneConsent05_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bergmann, J.; Bott, O.J.; Hoffmann, I. et al. (2005). \"An eConsent-based System Architecture Supporting Cooperation in Integrated Healthcare Networks\". <i>Studies in Health Technology and Informatics<\/i> <b>116<\/b>: 961\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16160382\" target=\"_blank\">16160382<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+eConsent-based+System+Architecture+Supporting+Cooperation+in+Integrated+Healthcare+Networks&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Bergmann%2C+J.%3B+Bott%2C+O.J.%3B+Hoffmann%2C+I.+et+al.&rft.au=Bergmann%2C+J.%3B+Bott%2C+O.J.%3B+Hoffmann%2C+I.+et+al.&rft.date=2005&rft.volume=116&rft.pages=961%E2%80%936&rft_id=info:pmid\/16160382&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BergmannAnEconsent07-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BergmannAnEconsent07_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bergmann, J.; Bott, O.J.; Pretschner, D.P. et al. (2007). \"An e-consent-based shared EHR system architecture for integrated healthcare networks\". <i>International Journal of Medical Informatics<\/i> <b>2\u20133<\/b>: 130\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2006.07.013\" target=\"_blank\">10.1016\/j.ijmedinf.2006.07.013<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16971171\" target=\"_blank\">16971171<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+e-consent-based+shared+EHR+system+architecture+for+integrated+healthcare+networks&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Bergmann%2C+J.%3B+Bott%2C+O.J.%3B+Pretschner%2C+D.P.+et+al.&rft.au=Bergmann%2C+J.%3B+Bott%2C+O.J.%3B+Pretschner%2C+D.P.+et+al.&rft.date=2007&rft.volume=2%E2%80%933&rft.pages=130%E2%80%936&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2006.07.013&rft_id=info:pmid\/16971171&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StaderiniAReq18-39\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-StaderiniAReq18_39-0\">39.0<\/a><\/sup> <sup><a href=\"#cite_ref-StaderiniAReq18_39-1\">39.1<\/a><\/sup> <sup><a href=\"#cite_ref-StaderiniAReq18_39-2\">39.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Staderini, M.; Schiavone, E.; Bondavalli, A. (2018). \"A Requirements-Driven Methodology for the Proper Selection and Configuration of Blockchains\". <i>Proceedings of the 2018 IEEE 37th Symposium on Reliable Distributed Systems<\/i>: 201\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FSRDS.2018.00031\" target=\"_blank\">10.1109\/SRDS.2018.00031<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Requirements-Driven+Methodology+for+the+Proper+Selection+and+Configuration+of+Blockchains&rft.jtitle=Proceedings+of+the+2018+IEEE+37th+Symposium+on+Reliable+Distributed+Systems&rft.aulast=Staderini%2C+M.%3B+Schiavone%2C+E.%3B+Bondavalli%2C+A.&rft.au=Staderini%2C+M.%3B+Schiavone%2C+E.%3B+Bondavalli%2C+A.&rft.date=2018&rft.pages=201%E2%80%936&rft_id=info:doi\/10.1109%2FSRDS.2018.00031&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarkakisSecurity19-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarkakisSecurity19_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Markakis, E.; Nikoloudakis, Y.; Pallis, E. et al. (2019). \"Security Assessment as a Service Cross-Layered System for the Adoption of Digital, Personalised and Trusted Healthcare\". <i>Proceedings of the 2019 IEEE 5th World Forum on Internet of Things<\/i>: 91\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FWF-IoT.2019.8767249\" target=\"_blank\">10.1109\/WF-IoT.2019.8767249<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Security+Assessment+as+a+Service+Cross-Layered+System+for+the+Adoption+of+Digital%2C+Personalised+and+Trusted+Healthcare&rft.jtitle=Proceedings+of+the+2019+IEEE+5th+World+Forum+on+Internet+of+Things&rft.aulast=Markakis%2C+E.%3B+Nikoloudakis%2C+Y.%3B+Pallis%2C+E.+et+al.&rft.au=Markakis%2C+E.%3B+Nikoloudakis%2C+Y.%3B+Pallis%2C+E.+et+al.&rft.date=2019&rft.pages=91%E2%80%934&rft_id=info:doi\/10.1109%2FWF-IoT.2019.8767249&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AngraalBlock17-41\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AngraalBlock17_41-0\">41.0<\/a><\/sup> <sup><a href=\"#cite_ref-AngraalBlock17_41-1\">41.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Angraal, S.; Krumholz, H.M.; Schulz, W.L. (2017). \"Blockchain Technology: Applications in Health Care\". <i>Circulation: Cardiovascular Quality and Outcomes<\/i> <b>10<\/b> (9): e003800. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1161%2FCIRCOUTCOMES.117.003800\" target=\"_blank\">10.1161\/CIRCOUTCOMES.117.003800<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28912202\" target=\"_blank\">28912202<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Blockchain+Technology%3A+Applications+in+Health+Care&rft.jtitle=Circulation%3A+Cardiovascular+Quality+and+Outcomes&rft.aulast=Angraal%2C+S.%3B+Krumholz%2C+H.M.%3B+Schulz%2C+W.L.&rft.au=Angraal%2C+S.%3B+Krumholz%2C+H.M.%3B+Schulz%2C+W.L.&rft.date=2017&rft.volume=10&rft.issue=9&rft.pages=e003800&rft_id=info:doi\/10.1161%2FCIRCOUTCOMES.117.003800&rft_id=info:pmid\/28912202&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ECCommiss15-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ECCommiss15_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">European Commission (29 July 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eur-lex.europa.eu\/eli\/dec\/2015\/1302\/oj\" target=\"_blank\">\"Commission Decision (EU) 2015\/1302 of 28 July 2015 on the identification of \u2018Integrating the Healthcare Enterprise\u2019 profiles for referencing in public procurement (Text with EEA relevance)\"<\/a>. <i>EUR-Lex<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eur-lex.europa.eu\/eli\/dec\/2015\/1302\/oj\" target=\"_blank\">https:\/\/eur-lex.europa.eu\/eli\/dec\/2015\/1302\/oj<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Commission+Decision+%28EU%29+2015%2F1302+of+28+July+2015+on+the+identification+of+%E2%80%98Integrating+the+Healthcare+Enterprise%E2%80%99+profiles+for+referencing+in+public+procurement+%28Text+with+EEA+relevance%29&rft.atitle=EUR-Lex&rft.aulast=European+Commission&rft.au=European+Commission&rft.date=29+July+2015&rft_id=https%3A%2F%2Feur-lex.europa.eu%2Feli%2Fdec%2F2015%2F1302%2Foj&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SpanakisSecure16-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SpanakisSecure16_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Spanakis, E.G.; Spanakis, M.; Karantanas, A. et al. (2016). \"Secure access to patient's health records using SpeechXRays a mutli-channel biometrics platform for user authentication\". <i>Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society<\/i>: 2541-2544. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FEMBC.2016.7591248\" target=\"_blank\">10.1109\/EMBC.2016.7591248<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28268840\" target=\"_blank\">28268840<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Secure+access+to+patient%27s+health+records+using+SpeechXRays+a+mutli-channel+biometrics+platform+for+user+authentication&rft.jtitle=Proceedings+of+the+38th+Annual+International+Conference+of+the+IEEE+Engineering+in+Medicine+and+Biology+Society&rft.aulast=Spanakis%2C+E.G.%3B+Spanakis%2C+M.%3B+Karantanas%2C+A.+et+al.&rft.au=Spanakis%2C+E.G.%3B+Spanakis%2C+M.%3B+Karantanas%2C+A.+et+al.&rft.date=2016&rft.pages=2541-2544&rft_id=info:doi\/10.1109%2FEMBC.2016.7591248&rft_id=info:pmid\/28268840&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GrammatikisSecuring19-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GrammatikisSecuring19_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Grammatikis, P.I.R.; Sarigiannidis, P.G.; Moscholios, I.D. (2019). \"Securing the Internet of Things: Challenges, threats and solutions\". <i>Internet of Things<\/i> <b>5<\/b>: 41\u201370. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.iot.2018.11.003\" target=\"_blank\">10.1016\/j.iot.2018.11.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Securing+the+Internet+of+Things%3A+Challenges%2C+threats+and+solutions&rft.jtitle=Internet+of+Things&rft.aulast=Grammatikis%2C+P.I.R.%3B+Sarigiannidis%2C+P.G.%3B+Moscholios%2C+I.D.&rft.au=Grammatikis%2C+P.I.R.%3B+Sarigiannidis%2C+P.G.%3B+Moscholios%2C+I.D.&rft.date=2019&rft.volume=5&rft.pages=41%E2%80%9370&rft_id=info:doi\/10.1016%2Fj.iot.2018.11.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PasqualeProtect14-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PasqualeProtect14_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pasquale, F.A.; Rgone, T.A. (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/digitalcommons.law.umaryland.edu\/fac_pubs\/1542\/\" target=\"_blank\">\"Protecting Health Privacy in an Era of Big Data Processing and Cloud Computing\"<\/a>. <i>Digital Commons<\/i>. University of Maryland Francis King Carey School of Law<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/digitalcommons.law.umaryland.edu\/fac_pubs\/1542\/\" target=\"_blank\">https:\/\/digitalcommons.law.umaryland.edu\/fac_pubs\/1542\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Protecting+Health+Privacy+in+an+Era+of+Big+Data+Processing+and+Cloud+Computing&rft.atitle=Digital+Commons&rft.aulast=Pasquale%2C+F.A.%3B+Rgone%2C+T.A.&rft.au=Pasquale%2C+F.A.%3B+Rgone%2C+T.A.&rft.date=2014&rft.pub=University+of+Maryland+Francis+King+Carey+School+of+Law&rft_id=https%3A%2F%2Fdigitalcommons.law.umaryland.edu%2Ffac_pubs%2F1542%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LinnBlock16-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LinnBlock16_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Linn, L.A.; Koo, M.B. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.healthit.gov\/sites\/default\/files\/11-74-ablockchainforhealthcare.pdf\" target=\"_blank\">\"Blockchain for health data and its potential use in health it and health care related research\"<\/a> (PDF). Office of the National Coordinator for Health Information Technology<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.healthit.gov\/sites\/default\/files\/11-74-ablockchainforhealthcare.pdf\" target=\"_blank\">https:\/\/www.healthit.gov\/sites\/default\/files\/11-74-ablockchainforhealthcare.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Blockchain+for+health+data+and+its+potential+use+in+health+it+and+health+care+related+research&rft.atitle=&rft.aulast=Linn%2C+L.A.%3B+Koo%2C+M.B.&rft.au=Linn%2C+L.A.%3B+Koo%2C+M.B.&rft.date=2016&rft.pub=Office+of+the+National+Coordinator+for+Health+Information+Technology&rft_id=https%3A%2F%2Fwww.healthit.gov%2Fsites%2Fdefault%2Ffiles%2F11-74-ablockchainforhealthcare.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MiyachiHOCBS21-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MiyachiHOCBS21_47-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Miyachi, K.; Mackey, T.K. (2021). \"hOCBS: A privacy-preserving blockchain framework for healthcare data leveraging an on-chain and off-chain system design\". <i>Information Processing & Management<\/i> <b>58<\/b> (3): 102535. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ipm.2021.102535\" target=\"_blank\">10.1016\/j.ipm.2021.102535<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=hOCBS%3A+A+privacy-preserving+blockchain+framework+for+healthcare+data+leveraging+an+on-chain+and+off-chain+system+design&rft.jtitle=Information+Processing+%26+Management&rft.aulast=Miyachi%2C+K.%3B+Mackey%2C+T.K.&rft.au=Miyachi%2C+K.%3B+Mackey%2C+T.K.&rft.date=2021&rft.volume=58&rft.issue=3&rft.pages=102535&rft_id=info:doi\/10.1016%2Fj.ipm.2021.102535&rfr_id=info:sid\/en.wikipedia.org:Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150615\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.195 seconds\nReal time usage: 4.372 seconds\nPreprocessor visited node count: 35421\/1000000\nPreprocessor generated node count: 39809\/1000000\nPost\u2010expand include size: 273894\/2097152 bytes\nTemplate argument size: 97264\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 741.862 1 - -total\n 90.20% 669.154 1 - Template:Reflist\n 76.12% 564.703 47 - Template:Citation\/core\n 65.41% 485.287 39 - Template:Cite_journal\n 12.47% 92.477 7 - Template:Cite_web\n 7.80% 57.849 63 - Template:Citation\/identifier\n 4.90% 36.354 1 - Template:Infobox_journal_article\n 4.69% 34.763 1 - Template:Infobox\n 3.88% 28.779 50 - Template:Citation\/make_link\n 3.24% 24.069 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12579-0!*!0!!en!5!* and timestamp 20210705150611 and revision id 43023\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange\">https:\/\/www.limswiki.org\/index.php\/Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","edf532278ba56a75b0d284a94702804c_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/69\/Fig1_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/32\/Fig2_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/92\/Fig3_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/92\/Fig4_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a2\/Fig5_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/fc\/Fig6_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b6\/Fig7_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/ad\/Fig8_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/a5\/Tab1_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/4b\/Fig9_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/ff\/Tab2_Spanakis_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/4\/47\/Tab3_Spanakis_FrontDigHlth2021_3.jpg"],"edf532278ba56a75b0d284a94702804c_timestamp":1625497571,"89396bdb73ec18118ddadafe0171ee20_type":"article","89396bdb73ec18118ddadafe0171ee20_title":"Security and privacy in cloud-based eHealth systems (Sivan and Zukarnain 2021)","89396bdb73ec18118ddadafe0171ee20_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Security_and_privacy_in_cloud-based_eHealth_systems","89396bdb73ec18118ddadafe0171ee20_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Security and privacy in cloud-based eHealth systems\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nSecurity and privacy in cloud-based eHealth systemsJournal\n \nSymmetryAuthor(s)\n \nSivan, Remya; Zukarnain, Zuriati A.Author affiliation(s)\n \nUniversity Putra MalaysiaPrimary contact\n \nEmail: gs59108 at student dot upm dot edu dot myYear published\n \n2021Volume and issue\n \n13(5)Article #\n \n742DOI\n \n10.3390\/sym13050742ISSN\n \n2073-8994Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/2073-8994\/13\/5\/742\/htmDownload\n \nhttps:\/\/www.mdpi.com\/2073-8994\/13\/5\/742\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methods and materials \n\n3.1 Research strategy \n3.2 Research questions \n\n\n4 Cloud computing and the state of the art \n\n4.1 Advantages of cloud-based eHealth systems \n\n\n5 Security issues of cloud-based eHealth systems \n6 Classification of security solutions in eHealth systems \n\n6.1 Cryptographic security \n\n6.1.1 PKE, SKE, and other encryption methods \n\n\n6.2 Access control management \n6.3 Integrated blockchain model \n6.4 Custom security solutions \n\n\n7 Proposed solutions for security in cloud-based eHealth systems \n8 Research issues and future directions \n9 Conclusions \n10 Acknowledgements \n\n10.1 Author contributions \n10.2 Funding \n10.3 Conflicts of interest \n\n\n11 References \n12 Notes \n\n\n\nAbstract \nCloud-based healthcare computing has changed the face of healthcare in many ways. The main advantages of cloud computing in healthcare are scalability of the required service and the provision to upscale or downsize the data storge, particularly in conjunction with approaches to artificial intelligence (AI) and machine learning. This paper examines various research studies to explore the utilization of intelligent techniques in health systems and mainly focuses on the security and privacy issues in the current technologies. Despite the various benefits related to cloud computing applications for healthcare, there are different types of management, technology handling, security measures, and legal issues to be considered and addressed. The key focus of this paper is to address the increased demand for cloud computing and its definition, technologies widely used in healthcare, their problems and possibilities, and the way protection mechanisms are organized and prepared when the company chooses to implement the latest evolving service model. As such, we sought out current literature on different approaches and mechanisms used in eHealth to deal with security and privacy issues. Some of these approaches have strengths and weaknesses. After selecting original articles, a literature review was carried out, and we identified several models adopted in their solutions. We arrived at the reviewed articles after comparing the models used.\nKeywords: eHealth, cloud computing, security, privacy in health systems\n\nIntroduction \nInnovative changes in science, technology, and the broad understanding of our universe have permitted the evolution of practical, progressive answers to enhance the nature of human existence. Researchers considering these innovative developments have identified and assessed wellbeing data from various sources to acquire actionable information and address issues concerning human wellbeing, particularly in the realm of healthcare. In this manner, the advancement of incorporated medical care innovations\u2014including technological innovations\u2014has the likelihood to enhance efficiency and improve understanding of the results at each level of the medical care framework. Long-term care (LTC) facilities are a crucial a part of the healthcare industry, providing care to the fastest-growing group of the population. However, the adoption of electronic health records (EHRs) in LTC facilities lags behind other areas of the health care industry.[1] The advancement of new electronic health (eHealth) application frameworks can take care of specific issues pertinent to conventional medical care frameworks by means of powerful patient wellbeing controls, pervasive information access, distant patient checking, quick clinical intercession, and decentralized electronic medical records (EMRs). These frameworks can oversee wellbeing data and patient information, upgrade personal satisfaction, increase coordinated effort, improve results, decrease expenses, and increase the general efficiency of electronic healthcare administrations.[2] For healthcare, EHRs are required to be shared among different healthcare organizations, medical drug manufacturers, pharmacists, medical insurance providers, researchers, and patients. This poses a significant challenge to keeping patients' sensitive data secure.[3] Eisenach highlights this in his depiction of eHealth as a tech industry that addresses the intersection of the internet, systems administration, and medical services, which has the potential to benefit the framework of clients and partners. The growing concept of eHealth involves the convergence of clinical informatics, general human wellbeing, and the wellbeing and the use of a secure internet, which helps drive the overall advancement of new innovation to tackle profound issues, drive down expenses, and improve understanding.[4]\nAlong these lines, models, gadgets, and frameworks associated with the internet of things (IoT) have become universal. Besides, the broad appropriation of IoT has harmonized with the improvement of interrelated, corresponding advances, for example, registering knowledge for medical care, business, industry, operational frameworks, etc. The efficient and safe usage of wellbeing data advancements, benefits, and all-encompassing eHealth frameworks requires exceedingly efficient and strong security frameworks to make such execution reasonable. The universality of IoT frameworks has driven the expansion of IoT innovation, remembering assorted designs for use by healthcare organizations. Connecting networks, gadgets, applications, and administration with IoT permits eHealth frameworks to share related data by utilizing the most recent IoT innovations.[5]\nIoT and distributed computing are progressive innovations that supplement each other\u2019s capacities when incorporated as flexible, versatile, and efficient tolerant medical service frameworks. The blend provides benefits, including simplicity of execution (contrasted with regular organizations), improved data security during correspondence, speedy access to records, and lower energy costs over customary modalities. Cloud-based IoT eHealth frameworks can significantly improve medical care benefits and advance persistent and efficient healthcare development. Within such frameworks, hidden IoT networks empower correspondence between clients, administrators, and workers, with clinical information being stored in the cloud. However, with new these improvements in distributed computing pushing healthcare beyond \"business as usual,\" a variety of data storage and security issues are revealed, requiring consideration.[6]\nDistributed and cloud computing are innovations that have and continue to transform the healthcare industry. Distributed computing can help improve a healthcare organization's adaptability, savvy, finances, data processing ability, and secure data sharing and distribution.[7] Cloud computing, when implemented well, should provide the ability to oversee applications and information, server accessibility, and end-used computing (EUC). Cloud computing should also ideally provide a complete, logical set of insights into a healthcare enterprise's infrastructure and end clients. By extension, cloud computing permits staff or workers to obtain and deal with their applications and information continuously on any gadget, from anywhere in the world with access to the internet.[8] However, we recognize that a coordinated effort to move information to distributed systems and the cloud brings serious security and protection worries for healthcare providers. As such, they must endeavor to fully address the effectiveness of, security of, and versatility associated with the introduction of cloud-based eHealth to the healthcare enterprise.\nThe fundamental motivation behind this paper is to examine the idea of cloud-based eHealth, the current utilization of eHealth in healthcare, and the challenges and solutions of cloud-based eHealth. This paper will walk through the state of distributed and cloud computing within the healthcare industry, along with the diverse opportunities and significant cloud-related security challenges associated with it. Additionally, we will discuss the plausible security arrangements for cloud-enabled eHealth.\n\nMethods and materials \nThe existing literature on eHealth security is difficult to survey. We originally identified and downloaded 40 papers from the ACM Digital Library, 57 articles from the IEEE Computer Society's Digital Library, and 43 from the IEEE Xplore library to provide a fair and calculable number of articles surveyed. Additional articles were also accessed from the specialized repositories of Springer, Elsevier, Science Direct, and MDPI. Few other papers from other sources were used as those sources were not as well-known and respected as from those cited. Of those papers analyzed, we identified 110 publications worthy of use in the face of conflicting evaluation models and methods used by many analysts. Due to the similarities found in the models obtained by some scientists, the analyzed papers were restricted to the momentum number.\nWe then directly analyzed the papers, looking at and dissecting the strengths of each of the approaches obtained in seeking a solution to the security problem in eHealth. Through this process, we identified numerous shortcomings of the various proposed techniques, which eventually indicated a way forward to mitigate eHealth security incidents. Improving healthcare outcomes using eHealth is a developing area at the crossroads of clinical informatics, general human wellbeing, and patient health data that is securely communicated and enhanced through the internet and related technologies.[9][10][11][12] The use of eHealth in the healthcare setting represents not olny a specialized turn of events from a more traditional approach, but also a perspective, a demeanor, and a duty for coordinated, national efforts to enhance patient care through the use of information and communication technologies locally, territorially, and internationally.[13]\n\nResearch strategy \nAs this was a comprehensive review, we aimed to update the findings of previous studies (54) regarding the same matter. We retrieved eligible studies from late 2013 to December 2020. The study selection searched for publications from the journal Symmetry, other journals in MDPI, IEEE Access, and other IEEE journals. The first step involved including the relevant articles with related keywords in the title or abstract based on the inclusion criteria. Papers not related to our study were excluded. The second step consisted of a full text screening of the relevant studies to select the most eligible articles.\n\nResearch questions \nWhen screening papers, we also wanted to ensure they would help us address four important questions:\n\nWhat is cloud computing and the state of the art of the cloud-based computing solutions commonly used in healthcare systems?\nWhat are the security concerns or challenges in cloud-based computing in healthcare systems?\nHow are the current cloud-based eHealth systems being protected?\nWhat is the best solution for security in cloud-based eHealth systems?\nCloud computing and the state of the art \nCloud computing\u2014a set of internet-based, always-on computing services provisioned on an as-needed or continuous basis\u2014is organized into three different service models: infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS).\nIaaS is a cloud service model that delivers computing resources, networking, and storage to consumers. It enables end users to upscale or downsize resources on an as-needed basis, reducing the need for compute upscaling, up-front capital expenditures, and unnecessary infrastructure.[14][15] PaaS is a cloud service model that provides an application development and deployment environment to comsumers. PaaS platforms handle the complexity around operating systems and servers and leaves application developers more time to focus their business requirements to the software they are developing. SaaS is a cloud service model that delivers software to customers with relative ease; all you need is an internet connection and a browser. This service model requires the cloud vendor to manage most of their technical issues, meaning customers need not heavily lean on their in-house IT expertise for hosting and maintaining the software.\nFigure 1 take a high-level look at these cloud infastructures, as compared with on-premises offerings. Light blue indicates customer responsibility, and dark blue indicates cloud provider responsibility.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Cloud service models and responsibilities, compared with on-premises systems. Light blue indicates customer responsibility, and dark blue indicates cloud provider responsibility.\n\n\n\nThe eHealth system is a newly developed space that contains electronic processes and communications. EHRs and EMRs act as a compilation of digital patient health data, containing demographic information, diagrams, patient medical information, patient medications, hospital or clinic reports, radiology photographs, billing information, and other sensitive patient information. Cloud computing offers a cost-effective way of effectively storing, processing, and updating that type of data with efficiency and quality.[16] Cloud computing also offers the advantage of access to hosted services from multiple locations around the world with by multiple users. \nSeparately, eHealth systems promise faster, robust, and sought-after access to medical records, fewer medical regulations, and improved health care quality, though they also should ideally maintain patient privacy, prevent improper authorization, and prevent misuse of sensitive patient data. When using these eHealth systems in the cloud, maintaining security and privacy are especially critical when sharing or accessing patient data. An overview of e-health structures in the scope of cloud computing is shown in Figure 2.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. eHealth architecture in the scope of cloud computing.\n\n\n\nThere are a number of deployment models for cloud computing, including public cloud, private cloud, and hybrid cloud. Public cloud deployments make resources available to private and public users via the internet as an ongoing or on-demand service, allowing clients to pay only for what they use or subscribe to. Instead of purchasing a physical server and networking equipment, a public cloud customer can purchase a virtual server and network that can be accessed from anywhere. The public cloud relies on a customized environment to provide corporate infrastructure expansion, allowing the company to host certain aspects of its infrastructure and services on virtual and identifiable third-party servers. Public cloud service providers have unique capabilities, and they offer excellent types of services and price models. However, companies considering migration to public cloud must carefully consider their options when it comes to choosing a provider, especially if there is a possibility of service being preemptively terminated in a long-term contract. Careful planning can help reduce the cost of monthly payments for cloud services, but organizations with unplanned or unexpected usage requirements may find it difficult to avoid spending large sums on public cloud services.[17][15]\nPrivate clouds are cloud installations used exclusively by a single organization and its authorized users, usually in the form of a single network or datacenter that provides services hosted to a specific group of people. Private clouds offer easy access and more control over security for the organizations using them. Although a private cloud is less expensive than using a public cloud, the cost is not easily measured. The development or growth of private cloud infrastructure will require the purchase of additional equipment. Similarly, when the need for a private cloud decreases, expensive resources and equipment are inevitably misused.[18][19]\nA hybrid cloud infrastructure is a combination of two or more integrated public and private clouds. In hybrid cloud infrastructure, the organization provides and manages their own resources within the private data center while also using additional outsourced public and private services\u2014such as VMware that works in tandem with Fortinet Networks\u2014to provide a hybrid cloud infrastructure that allows for increased organizational security on private and remote servers while the public cloud is largely secured by the cloud provider. Hybrid cloud is usually an excellent deal for those debating between a public cloud and a private cloud. The hybrid cloud environment allows organizations to take advantage of both types of cloud platforms and choose which cloud will provide the specific data needs. For example, the hybrid cloud provides another way to store sensitive company data and information, allowing the organization to maintain sensitive security configurations in the private cloud while conducting other business activities in the public cloud.[20][21]\n\nAdvantages of cloud-based eHealth systems \nThe more a healthcare center integrates system data and information via a global computer network such as the internet, the more global risk that is placed into the information network due to the expansion of breachable information access points. The need to access sensitive patient information should be limited to authorized users and protected from illegal users who may misuse that information for a variety of purposes. Identity-based encryption (IBE) has so far proven to be one of the best security solutions to protect eHealth record data.[22][23] The encryption algorithm deals with problems found in common cryptographic techniques using any thread as a public key. The system can further enhance the security of health records by adding authentication procedures to three connected servers. In this system, communication between three servers uses IBE to encrypt data, such that each server can perform the encryption and decryption process during the data exchange. Only servers with authorized IDs can access and extract health record data. Currently, test results show performance relative to the speed of the algorithm used in the system.[24][25]\nAs discussed here, and by Vilela et al.[26], cloud storage involves sending data over the internet to a remote cloud data center for storage. Cloud storage is delivered according to demand, capacity, and cost requirements, giving the customer control over how they invest in and manage their data storage infrastructure. This model has the advantage of lending speed, scale, and durability to an organization's eHealth data management.\nRunning eHealth systems in the cloud has additional advantages as well, including:\n\n Ease of access using a web browser and integrated single sign-on (SSO)\n No requirement for a virtual private network (VPN) to access other sites or networks\n Simplified data management and on-demand scalability\n No overhead cost to maintain the physical infrastructure\n No hardware-related post-warranty charges for the physical infrastructure\n Little to no additional power consumption\nMechanism commonly used in the cloud also lend to eHealth systems. For example, one of the major schemes in healthcare systems is attribute-based encryption for data. Encryption provides high-class access control for every user, as well as revocation, scalability, dynamic user management, and traceability. Users have high-grade login access, which can be integrated with two-factor authentication (2FA) as well as one-time passwords (OTP) for every access. Similarly, users\u2019 access can be revoked from a centralized management console at any point in time by the administrator. The most important advantage is that every access can be recorded and traced, and clipping can be implemented for each session.[27] Privileged access management (PAM) is another technology which can be used if there are many subsystems that a user needs to access at a single point in time.\nAnother cloud-based scheme for eHealth systems involves robust and secure access control, which resolves single-point performance bottleneck problems. This solves the majority of security access control issues, as the certificate needs to be trusted and issued from a certified hosting platform. The main disadvantage, however, is that there is no process or mechanism for attribute revocation.\n\nSecurity issues of cloud-based eHealth systems \nDespite the advantages cloud computing brings to eHealth, it still has its own security challenges which emerge from time to time due to inadequate compliance with proper security protocols and best practices. Some of those challenges include[28][29][30][31][32]:\n\n Confidentiality: Confidentiality is a process or mechanism of safeguarding patient health data from unauthorized access from public or internal users. Unauthorized access is dangerous and can potentially result in data leakage and can even cause serious damage to businesses. As the number of devices housing patient data increases, so does the potential threat (i.e., attack surface area) of exposing that data to external parties. Confidentiality is important in the healthcare industry, as the patient can be reluctant to give personal details to doctors if they are not confident the doctor's office will be able to keep that information confidential. By implementing access control and using encryption techniques, confidentiality can be achieved.\n Data integrity: Data integrity is required to ensure that data are not maliciously changed at any single point in time. This is especially important given that the Health Insurance Portability and Accountability Act (HIPAA) rules state that covered entities must implement procedures and policies to protect electronic healthcare information from improper destruction or alteration. Data integrity can be achieved by using a hashing mechanism or checksum for all data. This is most effectively done by implementing blockchain technology, as it is nearly impossible to change the hash of data with blockchain; it will change the entire chain if any of the hashes are changed.\n Data availability: Data availability says that information must be available at all times. Business-critical systems should be clustered or must have high availability to guarantee maximum uptime without service interruptions.\n Data access violations and breaches: These can negatively impact a business' public image and the the trust customers and partners have in the business. Intellectual property and sensitive data lost to these incidents can lead to other financial costs related to data forensics, communication to data owners, and more.\n Inappropriate updates and fixes: As cloud computing is a shared resource, any misconfiguration of the datacenter\u2014a common cloud challenge\u2014will potentially lead to compromising the customer data hosted within that datacenter.\n Lack of security technologies: The biggest challenge during the transition to cloud computing is the implementation of an appropriate security architecture to withstand cyberattacks. Unfortunately, this process remains a mystery for several organizations. Data are exposed to different threats when organizations assume that cloud migration is a \u201clift-and-shift\u201d endeavor of simply porting their existing IT stack and security controls to a cloud environment. A lack of knowledge of the shared security responsibility is also a contributing factor.\n Account hijacking: Whether by \"phishing,\" cloud system exploit, or direct attacks on a system, an internal user account can become compromised by an external actor, giving them access to sensitive information.\n Insider threat: In-house personnel can also perform malicious attacks on company assets by hacking servers or accessing and saving sensitive patient data, company emails, and more from insufficiently secured devices and programs.\n Unsecured APIs: Cloud computing providers develop a range of user software and APIs to allow customers to manage and interact with cloud services. The security and availability of standard cloud services are linked to the security of those APIs. Poorly designed APIs can lead to misuse or theft of information. Exposed, broken, and hacked APIs inevitably lead to realized data breaches.\nThese challenges and others highlight the need for healthcare systems to understand the security and regulatory requirements necessary for keeping data and information safe in the cloud.\n\nClassification of security solutions in eHealth systems \nCryptographic security \nThe continued advancement of information technology and data communications strengthens the exchange of highly sensitive medical information. Electronic health systems are widely used, and many medical facilities rely on the transmission and receipt of medical information online and across local networks. Over the years, many security systems have been introduced to monitor patient privacy and ensure the safety of interchangeable medical data. Cryptography is one of the primary techniques that provides security for eHealth systems.\nThe cryptographic method allows various computations to take place directly from encrypted data, without the need to define them. As such, encryption schemes with homomorphic structures can be useful in building confidentiality agreements, where confidential information remains protected not only during exchange and storage, but also processing.\nCryptography is a method of compiling and validating data, and then securely exchanging them, with the goal of keeping track of those data set up for them to investigate. Countless methods have been suggested to protect patient health care data. However, cryptographic methods can be seen in two forms: symmetric-key cryptography and asymmetric-key cryptography, where the former uses the same encryption and coding key while the latter uses different keys. These methods heavily rely on mechanisms such as public key encryption (PKE), symmetric key encryption (SKE), and a few native cryptographic devices used to protect the eHealth cloud. In the case of PKE, an open key or a pair of encryption and spelling keys are used; however, SKE-based methods use an equally shared private key as clearly defined in the PKE and SKE cryptography methods explained by Huang et al.[33]\n\nPKE, SKE, and other encryption methods \nIn PKE schemes, two different key sets are considered: public key and secret key. Pairs of private and public data encryption and encryption keys are use with PKE, whereas SKE-based approaches use the same shared private key. Other cryptographic schemes include several encryption schemes such as attribute-based encryption (ABE), searchable encryption (SE), proxy encryption, homomorphic encryption, and identity-based encryption (IBE).\nEncryption is vital to data security. Encryption locks down electronic information so that no one other than the holder of the key can access the information. Encryption even works if the organizer breaks the organization\u2019s security control on the information. The process involves encrypting data in one place, then transferring data over the network, and then deciphering data in the cloud, preventing data from being stolen during transfer.\nCryptography and encryption have had a profound effect on the security of eHealth systems.[34] There are some non-cryptographic methods that can also provide security, but these are not widely used as they provide partial security for the eHealth cloud under the security provided by crypto methods. Therefore, these systems are used with crypto methods in hybrid cloud systems. Other examples of encryption mechanisms include:\n\n Broadcast encryption programs: This type of \"stream\" encryption allows a mid-stream broadcast website to securerly transmit data to an associate while minimizing the high-speed communications required.[35]\n Qualified encryption: Privacy-based encryption can also be a form of public key cryptography in which the user\u2019s private key and the security unit deletion of text definitions is only possible if the set of user key symbols matches the text symbols.[36][37]\n Blockchain-based encryption: In the eHealth system, the verification, preservation, and synchronization of EMRs have always been a serious problem, and the random distribution of patient records will present various risks to the patient\u2019s privacy. Placing hash counts in electronic medical data and keeping the corresponding value in the blockchain ensure its consistency and integrity. Khezr et al. demonstrate this using blockchain and Ethereum.[38] In their work, details listed in clinical records were removed from the Ethereum blockchain, and a good agreement was submitted to the Ethereum blockchain to accept the viewing of word searches instead of the third-party person.[38]\n Searchable symmetric encryption (SSE): SSE allows users transferring data to the cloud to query the data entered without the risk of data loss. Although interest in SSE is growing, current research does not examine how structures under SSE are constructed and how they end up in various areas of the SSE system. Most applications use a file table, where the appropriate file size and sublinear search can be done using the inverted file list. Direct referrals can only be obtained using the direct reference.[39][40]\nAccess control management \nAnother type of security mechanism for cloud-based eHealth is the access control management (AAM) system, where the server identifies users and determines their access rights. Authorized users then receieve tokens to access their sensitive records stored in the cloud.[41][42]\n\nIntegrated blockchain model \nKurdi et al. proposed an integrated blockchain model of a fully independent blockchain (FPB) and consortium blockchain (CB) as a means to improve the time taken for data validation.[43] Here, FPB is used as an old database of health care facilities and CB is used to store medical data from all participating medical professionals, expanding the available options for deploying a eHealth cloud computing platform. \nWhile the confidentiality of data converted to the cloud between institutions is not processed, medical data remain sensitive. Electronic health applications include a shared distribution of digital information between providers and patients. This transfer is an end-to-end process that must be secured and managed regularly. As hackers continue to seek new vulnerabilities to take advantage of, any cloud-driven app should be updated frequently to stay ahead of threats. However, a balance must be struck between strong security, controlled with a simple yet effective authentication method, and making authorized access reasonably simple. Additionally, the level of security and the cost associated with it must be considered. Managing security for an EMR or some other eHealth system should be a relatively painless process that doesn't increase the premium of the solution.\n\nCustom security solutions \nTable 1 shows a variety of custom approaches to eHealth systems security, as described in the literature.\n\n\n\n\n\n\n\nTable 1. Custom approaches to eHealth systems security\n\n\nReference\n\nSecurity and privacy\n\neHealth\n\nCloud\n\nContribution\n\n\nGriebel et al. 2015[8]\n\n\u2713\n\n\u2713\n\n\u2713\n\nProposed secure personal eHealth system based on certificate CA to authorized users only\n\n\nVen\u010dkauskas et al. 2016[9]\n\n\u2713\n\n\u2713\n\n\u2713\n\nProposed a prototype-based personalised health monitoring application\n\n\nKhan 2020[10]\n\n\u2713\n\n\u2713\n\nx\n\nProposed a wearable IoT-enabled heart disease prediction system using the MDCNN classifier\n\n\nYang and Hara-Azumi 2020[11]\n\n\u2713\n\n\u2713\n\nx\n\nAddressed the development of IoT-based eHealth systems on both the hardware and software levels\n\n\nChenthara et al. 2019[16]\n\n\u2713\n\n\u2713\n\n\u2713\n\nExplored efficient comprehensive security mechanisms for EHR and the techniques to maintain the integrity and confidentiality of patients\u2019 information\n\n\nKim et al. 2019[18]\n\n\u2713\n\n\u2713\n\nx\n\nProposed a novel collaborative eHealth system, which supports multi-level privacy-preserving data sharing (MPPDS)\n\n\nZhang et al. 2018[44]\n\n\u2713\n\n\u2713\n\n\u2713\n\nProposed HealthDep, the first secure and efficient encrypted EMR deduplication scheme for cloud-assisted eHealth systems\n\n\nYeh et al. 2018[45]\n\n\u2713\n\n\u2713\n\n\u2713\n\nProposed a benchmark lossless method which is suitable for remoting health monitoring systems\n\n\nAvila et al. 2017[46]\n\n\u2713\n\n\u2713\n\n\u2713\n\nReviewed the applications developed in the health industry which are focused on patient care from home and implemented a service-oriented architecture (SOA) design\n\n\nGuo et al. 2014[47]\n\n\u2713\n\n\u2713\n\n\u2713\n\nProvided an answer for privacy-preserving knowledge storage by group action of PRF-based key management for unlikability, a research and access pattern concealment theme supported redundancy, and a secure assortment technique for privacy conserving keyword search\n\n\nLiu et al. 2018[48]\n\n\u2713\n\n\u2713\n\n\u2713\n\nProposed automatic internet of things (IoT) building blocks to exchange the information between devices and integrated cloud computing\n\n\n\nProposed solutions for security in cloud-based eHealth systems \nThe conventional approach to healthcare practices in the world has been changed by the growth of information and communication technologies. In the partial abandonment of paper-based medical procedures for more electronic procedures, this development is well noticed, particularly in most developed countries of the world. The need to federate and integrate electronic health information from medical research laboratories, hospitals, and health insurance companies has led to the development of a concept called electronic health or eHealth.[49] Simply put, eHealth can be defined as the use of information technology (IT) and e-commerce practices for health information processing, sharing, and management. However, these aspects can be difficult to manage in a cloud-based environment that enables the collaborative sharing of medical information across multiple administrative domains by multiple users.[50] \nTo be sure, cloud computing has many benefits, including the timely and seamless transfer and sharing of medical information across geophysical boundaries. Cloud computing also simplifies the management of infrastructure for healthcare providers and has provided them with ample opportunity to familiarize themselves with IT and cloud service providers. Various academic papers have established cloud computing as having numerous advantages, including scalability, cost-effectiveness, greater agility, and improved collaborative resource sharing.[51]\nHowever, despite the various advantages, there are also security and privacy challenges that urgently deserve the utmost attention for the realization of cloud computing's efficient and full-scale use.[52] On several occasions, cryptographic and non-cryptographic approaches have been used to ensure the preservation of cloud computing security and the privacy of health information. In addition, fine-grained as well as patient-centric access control systems are frequently used to achieve electronic health privacy. This paper has reviewed these different data security measures and discussed their strengths and weaknesses. Additionally, efforts have been made to provide examples of better options for securing eHealth information.\n\nResearch issues and future directions \nGoing forward, several privacy and security issues related to EHRs and their shared use must be addressed. EHR data are extremely sensitive, confidential, and in particular, are stored on third-party servers, leading to significant risks in terms of data privacy and security. Some of the major concerns that require further action include:\n\n developing processes and procedures on how to protect and secure EHR data stored in the cloud;\n using confidential EHR and other health care data via the cloud; and\n determining the best encryption system that can be used to maintain EHR data security in the cloud.\nThe above points are not limited to EHR data and extend to other eHealth systems. This highlights the need to use secure infrastructure in eHealth systems that will ensure the privacy and security of data and thus protect patient privacy and trust. The level of privacy required is high, making privacy considerations one of the most important aspects of the healthcare industry. Maintaining confidentiality through the sharing and used of medical records is essential to detecting and preventing fraud. Keeping track of data sources and programs is advisable.\nThe recommendations behind this research can be further enhanced by verifying user authenticity and privacy in cloud-based big data. A secure authentication method that incorporates tree-based hashing in the authorization structure, used for user authentication during multi-level configuration, resists fraudulent attacks and protects privacy. The authoritative structure of hierarchical attributes leads to a lot of confusion and significant problems. Artificial intelligence or big data analytics (BDA) could incorporate system design controls for each process. More recently, the usefulness of replica nodes has emerged. Leaked user data is displayed with random action keys in secure data connections. By using analytics tools, patient data can be merged into a single dashboard view, potentially improving patient outcomes.\n\nConclusions \nHealthcare has already adopted cloud-based solutions to solve multiple problems cost-effectively. While considering the future and growth of cloud-based healthcare services, we focused on the security of critical EHRs and other eHealth systems and proposed an identity-based secure and encrypted data-sharing technique. We have studied the existing PKE, IBE, IBBE, and ABE methods and then proposed multiple solutions to safeguard cloud-based data in eHealth systems. Appropriate security solutions should be developed and maintained to protect data security. The future of cloud-based eHealth services will be the integration of file-based and cloud-based applications that integrate a computer-based hybrid IT solution that measures the flexibility and scalability associated with cloud management and healthcare data security. This review highlights a comprehensive study of existing cloud-based eHealth solutions that incorporate cryptographic and non-cryptographic methods for protecting the privacy and security of digital data.\n\nAcknowledgements \nWe thank the anonymous reviewers for their valuable comments and suggestions, which helped us to improve the content of our paper.\n\nAuthor contributions \nR.S. and Z.A.Z.; methodology: R.S.; data curation: R.S.; resources: R.S.; writing\u2014original draft preparation: R.S.; writing\u2014review and editing: R.S. and Z.A.Z.; supervision. All authors have read and agreed to the published version of the manuscript.\n\nFunding \nThis research received no external funding.\n\nConflicts of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 Kruse, C.S.; Mileski, M.; Vijaykumar, A.G. et al. (2017). \"Impact of Electronic Health Records on Long-Term Care Facilities: Systematic Review\". JMIR Medical Informatics 5 (3): e35. doi:10.2196\/medinform.7958. PMC PMC5640822. PMID 28963091. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5640822 .   \n\n\u2191 Butpheng, C.; Yeh, K.-H.; Xiong, H. (2020). \"Security and Privacy in IoT-Cloud-Based e-Health Systems\u2014A Comprehensive Review\". Symmetry 12 (7): 1191. doi:10.3390\/sym12071191.   \n\n\u2191 Ismail, L.; Materwala, H. (2020). \"Blockchain Paradigm for Healthcare: Performance Evaluation\". Symmetry 12 (8): 1200. doi:10.3390\/sym12081200.   \n\n\u2191 Malluhi, Q.; Tran, V.D.; Trinh, V.C (2020). \"Decentralized Broadcast Encryption Schemes with Constant Size Ciphertext and Fast Decryption\". Symmetry 12 (6): 969. doi:10.3390\/sym12060969.   \n\n\u2191 Hassen, O.A.; Abdulhissein, A.A.; Darwish, S.M. et al. (2020). \"Towards a Secure Signature Scheme Based on Multimodal Biometric Technology: Application for IOT Blockchain Network\". Symmetry 12 (10): 1699. doi:10.3390\/sym12101699.   \n\n\u2191 Abdulghani, H.A.; Nijdam, N.A.; Collen, A. et al. (2019). \"A Study on Security and Privacy Guidelines, Countermeasures, Threats: IoT Data at Rest Perspective\". Symmetry 11 (6): 774. doi:10.3390\/sym11060774.   \n\n\u2191 Kang, J.; Chung, H.; Lee, J. et al. (2016). \"The Design and Analysis of a Secure Personal Healthcare System Based on Certificates\". Symmetry 8 (11): 129. doi:10.3390\/sym8110129.   \n\n\u2191 8.0 8.1 Griebel, L.; Prokosch, H.-U.; K\u00f6pcke, F. et al. (2015). \"A scoping review of cloud computing in healthcare\". BMC Medical Informatics and Decision Making 15: 17. doi:10.1186\/s12911-015-0145-7.   \n\n\u2191 9.0 9.1 Ven\u010dkauskas, A.; \u0160tuikys, V.; Toldinas, J. et al. (2016). \"A Model-Driven Framework to Develop Personalized Health Monitoring\". Symmetry 8 (7): 65. doi:10.3390\/sym8070065.   \n\n\u2191 10.0 10.1 Khan, M.A. (2020). \"An IoT Framework for Heart Disease Prediction Based on MDCNN Classifier\". IEEE Access 8: 34717\u201327. doi:10.1109\/ACCESS.2020.2974687.   \n\n\u2191 11.0 11.1 Yang, M.; Hara-Azumi, Y. (2020). \"Implementation of Lightweight eHealth Applications on a Low-Power Embedded Processor\". IEEE Access 8: 121724\u201332. doi:10.1109\/ACCESS.2020.3006901.   \n\n\u2191 Guo, L.; Li, Z.; Yau, W.-C. et al. (2020). \"A Decryptable Attribute-Based Keyword Search Scheme on eHealth Cloud in Internet of Things Platforms\". IEEE Access 8: 26107\u201318. doi:10.1109\/ACCESS.2020.2971088.   \n\n\u2191 Edemacu, K.; Park, H.K.; Jang, B. et al. (2019). \"Privacy Provision in Collaborative Ehealth With Attribute-Based Encryption: Survey, Challenges and Future Directions\". IEEE Access 7: 89614\u201489636. doi:10.1109\/ACCESS.2019.2925390.   \n\n\u2191 Ma, H.; Xie, Y.; Wang, J. et al. (2019). \"Revocable Attribute-Based Encryption Scheme With Efficient Deduplication for Ehealth Systems\". IEEE Access 7: 89205\u201317. doi:10.1109\/ACCESS.2019.2926627.   \n\n\u2191 15.0 15.1 Caiza, J.C.; Martin, Y.-S.; Guam\u00e1n, D.S. et al. (2019). \"Reusable Elements for the Systematic Design of Privacy-Friendly Information Systems: A Mapping Study\". IEEE Access 7: 66512\u201335. doi:10.1109\/ACCESS.2019.2918003.   \n\n\u2191 16.0 16.1 Chenthara, S.; Ahmed, K.; Wang, H. et al. (2019). \"Security and Privacy-Preserving Challenges of e-Health Solutions in Cloud Computing\". IEEE Access 7: 74361\u201382. doi:10.1109\/ACCESS.2019.2919982.   \n\n\u2191 Razaque, A.; Amsaad, F.; Khan, M.J. et al. (2019). \"Survey: Cybersecurity Vulnerabilities, Attacks and Solutions in the Medical Domain\". IEEE Access 7: 168774\u201397. doi:10.1109\/ACCESS.2019.2926627.   \n\n\u2191 18.0 18.1 Kim, J.W.; Edemacy, K.; Jang, B. (2019). \"MPPDS: Multilevel Privacy-Preserving Data Sharing in a Collaborative eHealth System\". IEEE Access 7: 109910\u201323. doi:10.1109\/ACCESS.2019.2933542.   \n\n\u2191 Bouras, M.A.; Lu, Q.; Zhang, F.; et al. (2020). \"Distributed Ledger Technology for eHealth Identity Privacy: State of The Art and Future Perspective\". Sensors 20 (2): 483. doi:10.3390\/s20020483. PMC PMC7013398. PMID 31952172. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7013398 .   \n\n\u2191 Seol, K.; Kim, Y.-G.; Lee, E. et al. (2018). \"Privacy-Preserving Attribute-Based Access Control Model for XML-Based Electronic Health Record System\". IEEE Access 6: 9114\u201428. doi:10.1109\/ACCESS.2018.2800288.   \n\n\u2191 Zhu, L.; Zhang, C.; Xu, C. et al. (2018). \"An Efficient and Privacy-Preserving Biometric Identification Scheme in Cloud Computing\". IEEE Access 6: 19025\u201333. doi:10.1109\/ACCESS.2018.2819166.   \n\n\u2191 Qadir, J.; Mujeeb-U-Rahman, M.; Rehmani, M.H. et al. (2017). \"IEEE Access Special Section Editorial: Health Informatics for the Developing World\". IEEE Access 5: 27818\u201323. doi:10.1109\/ACCESS.2017.2783118.   \n\n\u2191 Yeh, K.-H. (2016). \"A Secure IoT-Based Healthcare System With Body Sensor Networks\". IEEE Access 4: 10288\u201399. doi:10.1109\/ACCESS.2016.2638038.   \n\n\u2191 Islam, S.M.R.; Kwak, D.; Kabir, H. et al. (2015). \"The Internet of Things for Health Care: A Comprehensive Survey\". IEEE Access 3: 678\u2013708. doi:10.1109\/ACCESS.2015.2437951.   \n\n\u2191 Tahir, A.; Chen, F.; Khan, H.U. et al. (2020). \"A Systematic Review on Cloud Storage Mechanisms Concerning e-Healthcare Systems\". Sensors 20 (18): 5392. doi:10.3390\/s20185392. PMC PMC7570508. PMID 32967094. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7570508 .   \n\n\u2191 Vilela, P.H.; Rodrigues, J.J.P.C.; Righi, R.da R. et al. (2020). \"Looking at Fog Computing for E-Health through the Lens of Deployment Challenges and Applications\". Sensors 20 (9): 2553. doi:10.3390\/s20092553.   \n\n\u2191 Haque, R.U.; Hasan, A.S.M.T.; Jiang, Q. et al. (2020). \"Privacy-Preserving K-Nearest Neighbors Training over Blockchain-Based Encrypted Health Data\". Electronics 9 (12): 2096. doi:10.3390\/electronics9122096.   \n\n\u2191 Stamatellis, C.; Papadopoulos, P.; Pitropakis, N. et al. (2020). \"A Privacy-Preserving Healthcare Framework Using Hyperledger Fabric\". Sensors 20 (22): 6587. doi:10.3390\/s20226587. PMC PMC7698751. PMID 33218022. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7698751 .   \n\n\u2191 Yaqoob, T.; Abbas, H.; Atiquzzaman, M. et al. (2019). \"Security Vulnerabilities, Attacks, Countermeasures, and Regulations of Networked Medical Devices\u2014A Review\". IEEE Communications Surveys & Tutorials 21 (4): 3723\u201468. doi:10.1109\/COMST.2019.2914094.   \n\n\u2191 Seh, A.H.; Zarour, M.; Alenezi, M. et al. (2020). \"Healthcare Data Breaches: Insights and Implications\". Healthcare 8 (2): 133. doi:10.3390\/healthcare8020133.   \n\n\u2191 Liu, H.; Crespo, R.G.; Martinez, O.S. (2020). \"Enhancing Privacy and Data Security across Healthcare Applications Using Blockchain and Distributed Ledger Concepts\". Healthcare 8 (3): 243. doi:10.3390\/healthcare8030243.   \n\n\u2191 Dang, L.M.; Piran, M.J.; Han, D. et al. (2019). \"A Survey on Internet of Things and Cloud Computing for Healthcare\". Electronics 8 (7): 768. doi:10.3390\/electronics8070768.   \n\n\u2191 Huang, Q.; Yue, W.; He, Y. et al. (2018). \"Secure Identity-Based Data Sharing and Profile Matching for Mobile Healthcare Social Networks in Cloud Computing\". IEEE Access 6: 36584\u201394. doi:10.1109\/ACCESS.2018.2852784.   \n\n\u2191 Celesti, A.; Fazio, M.; M\u00e1rquez, F.G. et al. (2019). \"How to Develop IoT Cloud e-Health Systems Based on FIWARE: A Lesson Learnt\". Journal of Sensor and Actuator Networks 8 (1): 7. doi:10.3390\/jsan8010007.   \n\n\u2191 Schiza, E.C.; Kyprianou, T.C.; Petkov, N. et al. (2019). \"Proposal for an eHealth Based Ecosystem Serving National Healthcare\". IEEE Journal of Biomedical and Health Informatics 23 (3): 1346\u201357. doi:10.1109\/JBHI.2018.2834230. PMID 29993757.   \n\n\u2191 Lin, H.-Y.; Jiang, Y.-R. (2021). \"A Multi-User Ciphertext Policy Attribute-Based Encryption Scheme with Keyword Search for Medical Cloud System\". Applied Sciences 11 (1): 63. doi:10.3390\/app11010063.   \n\n\u2191 Mehmood, Z.; Ghani, A.; Chen, G. et al. (2019). \"Authentication and Secure Key Management in E-Health Services: A Robust and Efficient Protocol Using Biometrics\". IEEE Access 7: 113385\u201397. doi:10.1109\/ACCESS.2019.2935313.   \n\n\u2191 38.0 38.1 Khezr, S.; Moniruzzaman, M.; Yassine, A. et al. (2019). \"Blockchain Technology in Healthcare: A Comprehensive Review and Directions for Future Research\". Applied Sciences 9 (9): 1736. doi:10.3390\/app9091736.   \n\n\u2191 Li, H.; Yang, Y.; Dai, Y. et al. (2017). \"Achieving Secure and Efficient Dynamic Searchable Symmetric Encryption over Medical Cloud Data\". IEEE Transactions on Cloud Computing 8 (2): 484\u201394. doi:10.1109\/TCC.2017.2769645.   \n\n\u2191 Zhang, R.; Xue, R.; Liu, L. (2017). \"Searchable Encryption for Healthcare Clouds: A Survey\". IEEE Transactions on Services Computing 11 (6): 978\u201396. doi:10.1109\/TSC.2017.2762296.   \n\n\u2191 Edemacu, K.; Jang, B.; Kim, J.W. (2020). \"Collaborative Ehealth Privacy and Security: An Access Control With Attribute Revocation Based on OBDD Access Structure\". IEEE Journal of Biomedical and Health Informatics 24 (10): 2960-2972. doi:10.1109\/JBHI.2020.2973713. PMID 32071017.   \n\n\u2191 Tong, Y.; Sun, J.; Chow, S.S.M. et al. (2014). \"Cloud-assisted mobile-access of health data with privacy and auditability\". IEEE Journal of Biomedical and Health Informatics 18 (2): 419\u201329. doi:10.1109\/JBHI.2013.2294932. PMID 24608047.   \n\n\u2191 Kurdi, H.; Alsalmah, S.; Alatawi, A. et al. (2019). \"HealthyBroker: A Trustworthy Blockchain-Based Multi-Cloud Broker for Patient-Centered eHealth Services\". Electronics 8 (6): 602. doi:10.3390\/electronics8060602.   \n\n\u2191 Zhang, Y.; Xu, C.; Li, H. et al. (2018). \"HealthDep: An Efficient and Secure Deduplication Scheme for Cloud-Assisted eHealth Systems\". IEEE Transactions on Industrial Informatics 14 (9): 4101\u201312. doi:10.1109\/TII.2018.2832251.   \n\n\u2191 Yeh, L.-Y.; Chiang, P.-Y.; Tsai, Y.-L. et al. (2018). \"Cloud-Based Fine-Grained Health Information Access Control Framework for Lightweight IoT Devices with Dynamic Auditing and Attribute Revocation\". IEEE Transactions on Cloud Computing 6 (2): 532\u201344. doi:10.1109\/TCC.2015.2485199.   \n\n\u2191 Avila, K.; Sanartin, P.; Jabba, D. et al. (2017). \"Applications Based on Service-Oriented Architecture (SOA) in the Field of Home Healthcare\". Sensors 17 (8): 1703. doi:10.3390\/s17081703.   \n\n\u2191 Guo, L.; Zhang, C.; Sun, J. et al. (2014). \"A Privacy-Preserving Attribute-Based Authentication System for Mobile Health Networks\". IEEE Transactions on Mobile Computing 13 (9): 1927\u201341. doi:10.1109\/TMC.2013.84.   \n\n\u2191 Liu, Y.; Zhang, Y.; Ling, J. et al. (2018). \"Secure and fine-grained access control on e-healthcare records in mobile cloud computing\". Future Generation Computer Systems 78 (3): 1020\u201326. doi:10.1016\/j.future.2016.12.027.   \n\n\u2191 Elgendi, M.; Al-Ali, A.; Mohamed, A. et al. (2018). \"Improving Remote Health Monitoring: A Low-Complexity ECG Compression Approach\". Diagnostics 8 (1): 10. doi:10.3390\/diagnostics8010010.   \n\n\u2191 Liagkou, V.; Kavvadas, V.; Chronopoulos, S.K. et al. (2019). \"Attack Detection for Healthcare Monitoring Systems Using Mechanical Learning in Virtual Private Networks over Optical Transport Layer Architecture\". Computation 7 (2): 24. doi:10.3390\/computation7020024.   \n\n\u2191 AbuKhousa, E.; Mohamed, N.; Al-Jaroodi, J. (2012). \"e-Health Cloud: Opportunities and Challenges\". Future Internet 4 (3): 621\u201345. doi:10.3390\/fi4030621.   \n\n\u2191 Sahmim, S.; Gharsellaoui, H. (2017). \"Privacy and Security in Internet-based Computing: Cloud Computing, Internet of Things, Cloud of Things: A review\". Procedia Computer Science 112: 1516\u201322. doi:10.1016\/j.procs.2017.08.050.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. However, extensive grammar, punctuation, and idea organization and clean up was required to improve the readability of the original article. In some cases important information was missing from the references, and that information was added. The original citation seven (Huh 2018) was omitted in this version as it appeared to have no connection to the topic cited. A paragraph appearing in the original introduction involving \"mists\" and service level agreements also had no bearing to the paper and appears to have been accidentally included in the original; it is omitted for this version. The original includes material from Azeez and Van der Vyver 2019 in the \"Proposed solutions for security in cloud-based eHealth systems\" section; that content was omitted for this version as it appears inadvertent or plagiarized. Finally, the Discussion in the original is a complete disaster, referencing numerous articles that aren't even discussed or cited (e.g., a mention of Rahmani et al.'s UT-Gate, as well as more almost-plagiarized content from Azeez and Van der Vyver 2019). The discussion was completely removed for this version as it too appears to be inadvertent or plagiarized content.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Security_and_privacy_in_cloud-based_eHealth_systems\">https:\/\/www.limswiki.org\/index.php\/Journal:Security_and_privacy_in_cloud-based_eHealth_systems<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on cloud computingLIMSwiki journal articles on health informaticsLIMSwiki journal articles on internet of things\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 2 June 2021, at 22:36.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 211 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","89396bdb73ec18118ddadafe0171ee20_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Security_and_privacy_in_cloud-based_eHealth_systems skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Security and privacy in cloud-based eHealth systems<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">Cloud-based<\/a> healthcare computing has changed the face of healthcare in many ways. The main advantages of cloud computing in healthcare are scalability of the required service and the provision to upscale or downsize the data storge, particularly in conjunction with approaches to <a href=\"https:\/\/www.limswiki.org\/index.php\/Artificial_intelligence\" title=\"Artificial intelligence\" class=\"wiki-link\" data-key=\"0c45a597361ca47e1cd8112af676276e\">artificial intelligence<\/a> (AI) and <a href=\"https:\/\/www.limswiki.org\/index.php\/Machine_learning\" title=\"Machine learning\" class=\"wiki-link\" data-key=\"79aab39cfa124c958cd1dbcab3dde122\">machine learning<\/a>. This paper examines various research studies to explore the utilization of intelligent techniques in health systems and mainly focuses on the <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_security\" title=\"Information security\" class=\"wiki-link\" data-key=\"9eff362d944224ff1d4ffe3a149d7cff\">security<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_privacy\" title=\"Information privacy\" class=\"wiki-link\" data-key=\"185f6d9f874e48914b5789317408f782\">privacy<\/a> issues in the current technologies. Despite the various benefits related to cloud computing applications for healthcare, there are different types of management, technology handling, security measures, and legal issues to be considered and addressed. The key focus of this paper is to address the increased demand for cloud computing and its definition, technologies widely used in healthcare, their problems and possibilities, and the way protection mechanisms are organized and prepared when the company chooses to implement the latest evolving service model. As such, we sought out current literature on different approaches and mechanisms used in <a href=\"https:\/\/www.limswiki.org\/index.php\/EHealth\" title=\"EHealth\" class=\"wiki-link\" data-key=\"39df6aac1fbe4ad737280794f3a81d80\">eHealth<\/a> to deal with security and privacy issues. Some of these approaches have strengths and weaknesses. After selecting original articles, a literature review was carried out, and we identified several models adopted in their solutions. We arrived at the reviewed articles after comparing the models used.\n<\/p><p><b>Keywords<\/b>: eHealth, cloud computing, security, privacy in health systems\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Innovative changes in science, technology, and the broad understanding of our universe have permitted the evolution of practical, progressive answers to enhance the nature of human existence. Researchers considering these innovative developments have identified and assessed wellbeing data from various sources to acquire actionable information and address issues concerning human wellbeing, particularly in the realm of healthcare. In this manner, the advancement of incorporated medical care innovations\u2014including technological innovations\u2014has the likelihood to enhance efficiency and improve understanding of the results at each level of the medical care framework. Long-term care (LTC) facilities are a crucial a part of the healthcare industry, providing care to the fastest-growing group of the population. However, the adoption of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a> (EHRs) in LTC facilities lags behind other areas of the health care industry.<sup id=\"rdp-ebb-cite_ref-KruseImpact17_1-0\" class=\"reference\"><a href=\"#cite_note-KruseImpact17-1\">[1]<\/a><\/sup> The advancement of new electronic health (<a href=\"https:\/\/www.limswiki.org\/index.php\/EHealth\" title=\"EHealth\" class=\"wiki-link\" data-key=\"39df6aac1fbe4ad737280794f3a81d80\">eHealth<\/a>) application frameworks can take care of specific issues pertinent to conventional medical care frameworks by means of powerful patient wellbeing controls, pervasive information access, distant patient checking, quick clinical intercession, and decentralized <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical records<\/a> (EMRs). These frameworks can oversee wellbeing data and patient <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, upgrade personal satisfaction, increase coordinated effort, improve results, decrease expenses, and increase the general efficiency of electronic healthcare administrations.<sup id=\"rdp-ebb-cite_ref-ButphengSecur20_2-0\" class=\"reference\"><a href=\"#cite_note-ButphengSecur20-2\">[2]<\/a><\/sup> For healthcare, EHRs are required to be shared among different healthcare organizations, medical drug manufacturers, pharmacists, medical insurance providers, researchers, and patients. This poses a significant challenge to keeping patients' sensitive data secure.<sup id=\"rdp-ebb-cite_ref-IsmailBlock20_3-0\" class=\"reference\"><a href=\"#cite_note-IsmailBlock20-3\">[3]<\/a><\/sup> Eisenach highlights this in his depiction of eHealth as a tech industry that addresses the intersection of the internet, systems administration, and medical services, which has the potential to benefit the framework of clients and partners. The growing concept of eHealth involves the convergence of <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">clinical informatics<\/a>, general human wellbeing, and the wellbeing and the use of a secure internet, which helps drive the overall advancement of new innovation to tackle profound issues, drive down expenses, and improve understanding.<sup id=\"rdp-ebb-cite_ref-MalluhiDecent20_4-0\" class=\"reference\"><a href=\"#cite_note-MalluhiDecent20-4\">[4]<\/a><\/sup>\n<\/p><p>Along these lines, models, gadgets, and frameworks associated with the <a href=\"https:\/\/www.limswiki.org\/index.php\/Internet_of_things\" title=\"Internet of things\" class=\"wiki-link\" data-key=\"13e0b826fa1770fe4bea72e3cb942f0f\">internet of things<\/a> (IoT) have become universal. Besides, the broad appropriation of IoT has harmonized with the improvement of interrelated, corresponding advances, for example, registering knowledge for medical care, business, industry, operational frameworks, etc. The efficient and safe usage of wellbeing data advancements, benefits, and all-encompassing eHealth frameworks requires exceedingly efficient and strong security frameworks to make such execution reasonable. The universality of IoT frameworks has driven the expansion of IoT innovation, remembering assorted designs for use by healthcare organizations. Connecting networks, gadgets, applications, and administration with IoT permits eHealth frameworks to share related data by utilizing the most recent IoT innovations.<sup id=\"rdp-ebb-cite_ref-HassenTowards20_5-0\" class=\"reference\"><a href=\"#cite_note-HassenTowards20-5\">[5]<\/a><\/sup>\n<\/p><p>IoT and distributed computing are progressive innovations that supplement each other\u2019s capacities when incorporated as flexible, versatile, and efficient tolerant medical service frameworks. The blend provides benefits, including simplicity of execution (contrasted with regular organizations), improved data security during correspondence, speedy access to records, and lower energy costs over customary modalities. <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">Cloud-based<\/a> IoT eHealth frameworks can significantly improve medical care benefits and advance persistent and efficient healthcare development. Within such frameworks, hidden IoT networks empower correspondence between clients, administrators, and workers, with clinical information being stored in the cloud. However, with new these improvements in distributed computing pushing healthcare beyond \"business as usual,\" a variety of data storage and security issues are revealed, requiring consideration.<sup id=\"rdp-ebb-cite_ref-AbdulghaniAStudy19_6-0\" class=\"reference\"><a href=\"#cite_note-AbdulghaniAStudy19-6\">[6]<\/a><\/sup>\n<\/p><p>Distributed and cloud computing are innovations that have and continue to transform the healthcare industry. Distributed computing can help improve a healthcare organization's adaptability, savvy, finances, data processing ability, and secure data sharing and distribution.<sup id=\"rdp-ebb-cite_ref-KangTheDesign16_7-0\" class=\"reference\"><a href=\"#cite_note-KangTheDesign16-7\">[7]<\/a><\/sup> Cloud computing, when implemented well, should provide the ability to oversee applications and information, server accessibility, and end-used computing (EUC). Cloud computing should also ideally provide a complete, logical set of insights into a healthcare enterprise's infrastructure and end clients. By extension, cloud computing permits staff or workers to obtain and deal with their applications and information continuously on any gadget, from anywhere in the world with access to the internet.<sup id=\"rdp-ebb-cite_ref-GriebelAScoping15_8-0\" class=\"reference\"><a href=\"#cite_note-GriebelAScoping15-8\">[8]<\/a><\/sup> However, we recognize that a coordinated effort to move information to distributed systems and the cloud brings serious security and protection worries for healthcare providers. As such, they must endeavor to fully address the effectiveness of, security of, and versatility associated with the introduction of cloud-based eHealth to the healthcare enterprise.\n<\/p><p>The fundamental motivation behind this paper is to examine the idea of cloud-based eHealth, the current utilization of eHealth in healthcare, and the challenges and solutions of cloud-based eHealth. This paper will walk through the state of distributed and cloud computing within the healthcare industry, along with the diverse opportunities and significant cloud-related security challenges associated with it. Additionally, we will discuss the plausible security arrangements for cloud-enabled eHealth.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods_and_materials\">Methods and materials<\/span><\/h2>\n<p>The existing literature on eHealth security is difficult to survey. We originally identified and downloaded 40 papers from the ACM Digital Library, 57 articles from the IEEE Computer Society's Digital Library, and 43 from the IEEE Xplore library to provide a fair and calculable number of articles surveyed. Additional articles were also accessed from the specialized repositories of Springer, Elsevier, Science Direct, and MDPI. Few other papers from other sources were used as those sources were not as well-known and respected as from those cited. Of those papers analyzed, we identified 110 publications worthy of use in the face of conflicting evaluation models and methods used by many analysts. Due to the similarities found in the models obtained by some scientists, the analyzed papers were restricted to the momentum number.\n<\/p><p>We then directly analyzed the papers, looking at and dissecting the strengths of each of the approaches obtained in seeking a solution to the security problem in eHealth. Through this process, we identified numerous shortcomings of the various proposed techniques, which eventually indicated a way forward to mitigate eHealth security incidents. Improving healthcare outcomes using eHealth is a developing area at the crossroads of clinical informatics, general human wellbeing, and patient health data that is securely communicated and enhanced through the internet and related technologies.<sup id=\"rdp-ebb-cite_ref-Ven.C4.8DkauskasAModel16_9-0\" class=\"reference\"><a href=\"#cite_note-Ven.C4.8DkauskasAModel16-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KhanAnIoT20_10-0\" class=\"reference\"><a href=\"#cite_note-KhanAnIoT20-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-YangImplem20_11-0\" class=\"reference\"><a href=\"#cite_note-YangImplem20-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GuoADecrypt20_12-0\" class=\"reference\"><a href=\"#cite_note-GuoADecrypt20-12\">[12]<\/a><\/sup> The use of eHealth in the healthcare setting represents not olny a specialized turn of events from a more traditional approach, but also a perspective, a demeanor, and a duty for coordinated, national efforts to enhance patient care through the use of information and communication technologies locally, territorially, and internationally.<sup id=\"rdp-ebb-cite_ref-EdemacuPrivacy19_13-0\" class=\"reference\"><a href=\"#cite_note-EdemacuPrivacy19-13\">[13]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Research_strategy\">Research strategy<\/span><\/h3>\n<p>As this was a comprehensive review, we aimed to update the findings of previous studies (54) regarding the same matter. We retrieved eligible studies from late 2013 to December 2020. The study selection searched for publications from the journal <i>Symmetry<\/i>, other journals in MDPI, <i>IEEE Access<\/i>, and other IEEE journals. The first step involved including the relevant articles with related keywords in the title or abstract based on the inclusion criteria. Papers not related to our study were excluded. The second step consisted of a full text screening of the relevant studies to select the most eligible articles.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Research_questions\">Research questions<\/span><\/h3>\n<p>When screening papers, we also wanted to ensure they would help us address four important questions:\n<\/p>\n<ol><li>What is cloud computing and the state of the art of the cloud-based computing solutions commonly used in healthcare systems?<\/li>\n<li>What are the security concerns or challenges in cloud-based computing in healthcare systems?<\/li>\n<li>How are the current cloud-based eHealth systems being protected?<\/li>\n<li>What is the best solution for security in cloud-based eHealth systems?<\/li><\/ol>\n<h2><span class=\"mw-headline\" id=\"Cloud_computing_and_the_state_of_the_art\">Cloud computing and the state of the art<\/span><\/h2>\n<p>Cloud computing\u2014a set of internet-based, always-on computing services provisioned on an as-needed or continuous basis\u2014is organized into three different service models: <a href=\"https:\/\/www.limswiki.org\/index.php\/Infrastructure_as_a_service\" title=\"Infrastructure as a service\" class=\"wiki-link\" data-key=\"70b93c66f23363688d45f0d354e5c032\">infrastructure as a service<\/a> (IaaS), <a href=\"https:\/\/www.limswiki.org\/index.php\/Platform_as_a_service\" title=\"Platform as a service\" class=\"wiki-link\" data-key=\"abad55890b97c6a6153458ad3d62762f\">platform as a service<\/a> (PaaS), and <a href=\"https:\/\/www.limswiki.org\/index.php\/Software_as_a_service\" title=\"Software as a service\" class=\"wiki-link\" data-key=\"ae8c8a7cd5ee1a264f4f0bbd4a4caedd\">software as a service<\/a> (SaaS).\n<\/p><p>IaaS is a cloud service model that delivers computing resources, networking, and storage to consumers. It enables end users to upscale or downsize resources on an as-needed basis, reducing the need for compute upscaling, up-front capital expenditures, and unnecessary infrastructure.<sup id=\"rdp-ebb-cite_ref-MaRevoc19_14-0\" class=\"reference\"><a href=\"#cite_note-MaRevoc19-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CaizaReusable19_15-0\" class=\"reference\"><a href=\"#cite_note-CaizaReusable19-15\">[15]<\/a><\/sup> PaaS is a cloud service model that provides an application development and deployment environment to comsumers. PaaS platforms handle the complexity around operating systems and servers and leaves application developers more time to focus their business requirements to the software they are developing. SaaS is a cloud service model that delivers software to customers with relative ease; all you need is an internet connection and a browser. This service model requires the cloud vendor to manage most of their technical issues, meaning customers need not heavily lean on their in-house IT expertise for hosting and maintaining the software.\n<\/p><p>Figure 1 take a high-level look at these cloud infastructures, as compared with on-premises offerings. Light blue indicates customer responsibility, and dark blue indicates cloud provider responsibility.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Sivan_Symmetry21_13-5.png\" class=\"image wiki-link\" data-key=\"4253593883614a9eee1452fb3737e472\"><img alt=\"Fig1 Sivan Symmetry21 13-5.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/29\/Fig1_Sivan_Symmetry21_13-5.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Cloud service models and responsibilities, compared with on-premises systems. Light blue indicates customer responsibility, and dark blue indicates cloud provider responsibility.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The eHealth system is a newly developed space that contains electronic processes and communications. EHRs and EMRs act as a compilation of digital patient health data, containing demographic information, diagrams, patient medical information, patient medications, <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> or clinic reports, radiology photographs, billing information, and other sensitive patient information. Cloud computing offers a cost-effective way of effectively storing, processing, and updating that type of data with efficiency and quality.<sup id=\"rdp-ebb-cite_ref-ChentharaSecurity19_16-0\" class=\"reference\"><a href=\"#cite_note-ChentharaSecurity19-16\">[16]<\/a><\/sup> Cloud computing also offers the advantage of access to hosted services from multiple locations around the world with by multiple users. \n<\/p><p>Separately, eHealth systems promise faster, robust, and sought-after access to medical records, fewer medical regulations, and improved health care quality, though they also should ideally maintain patient privacy, prevent improper authorization, and prevent misuse of sensitive patient data. When using these eHealth systems in the cloud, maintaining security and privacy are especially critical when sharing or accessing patient data. An overview of e-health structures in the scope of cloud computing is shown in Figure 2.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Sivan_Symmetry21_13-5.png\" class=\"image wiki-link\" data-key=\"700bc53268c9b5d3f8c741772e36c518\"><img alt=\"Fig2 Sivan Symmetry21 13-5.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/cd\/Fig2_Sivan_Symmetry21_13-5.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> eHealth architecture in the scope of cloud computing.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>There are a number of deployment models for cloud computing, including public cloud, private cloud, and hybrid cloud. Public cloud deployments make resources available to private and public users via the internet as an ongoing or on-demand service, allowing clients to pay only for what they use or subscribe to. Instead of purchasing a physical server and networking equipment, a public cloud customer can purchase a virtual server and network that can be accessed from anywhere. The public cloud relies on a customized environment to provide corporate infrastructure expansion, allowing the company to host certain aspects of its infrastructure and services on virtual and identifiable third-party servers. Public cloud service providers have unique capabilities, and they offer excellent types of services and price models. However, companies considering migration to public cloud must carefully consider their options when it comes to choosing a provider, especially if there is a possibility of service being preemptively terminated in a long-term contract. Careful planning can help reduce the cost of monthly payments for cloud services, but organizations with unplanned or unexpected usage requirements may find it difficult to avoid spending large sums on public cloud services.<sup id=\"rdp-ebb-cite_ref-RazaqueSurv19_17-0\" class=\"reference\"><a href=\"#cite_note-RazaqueSurv19-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CaizaReusable19_15-1\" class=\"reference\"><a href=\"#cite_note-CaizaReusable19-15\">[15]<\/a><\/sup>\n<\/p><p>Private clouds are cloud installations used exclusively by a single organization and its authorized users, usually in the form of a single network or datacenter that provides services hosted to a specific group of people. Private clouds offer easy access and more control over security for the organizations using them. Although a private cloud is less expensive than using a public cloud, the cost is not easily measured. The development or growth of private cloud infrastructure will require the purchase of additional equipment. Similarly, when the need for a private cloud decreases, expensive resources and equipment are inevitably misused.<sup id=\"rdp-ebb-cite_ref-KimMPPDS19_18-0\" class=\"reference\"><a href=\"#cite_note-KimMPPDS19-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BourasDistrib20_19-0\" class=\"reference\"><a href=\"#cite_note-BourasDistrib20-19\">[19]<\/a><\/sup>\n<\/p><p>A hybrid cloud infrastructure is a combination of two or more integrated public and private clouds. In hybrid cloud infrastructure, the organization provides and manages their own resources within the private data center while also using additional outsourced public and private services\u2014such as VMware that works in tandem with Fortinet Networks\u2014to provide a hybrid cloud infrastructure that allows for increased organizational security on private and remote servers while the public cloud is largely secured by the cloud provider. Hybrid cloud is usually an excellent deal for those debating between a public cloud and a private cloud. The hybrid cloud environment allows organizations to take advantage of both types of cloud platforms and choose which cloud will provide the specific data needs. For example, the hybrid cloud provides another way to store sensitive company data and information, allowing the organization to maintain sensitive security configurations in the private cloud while conducting other business activities in the public cloud.<sup id=\"rdp-ebb-cite_ref-SeolPrivacy18_20-0\" class=\"reference\"><a href=\"#cite_note-SeolPrivacy18-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZhuAnEff18_21-0\" class=\"reference\"><a href=\"#cite_note-ZhuAnEff18-21\">[21]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Advantages_of_cloud-based_eHealth_systems\">Advantages of cloud-based eHealth systems<\/span><\/h3>\n<p>The more a healthcare center integrates system data and information via a global computer network such as the internet, the more global risk that is placed into the information network due to the expansion of breachable information access points. The need to access sensitive patient information should be limited to authorized users and protected from illegal users who may misuse that information for a variety of purposes. Identity-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Encryption\" title=\"Encryption\" class=\"wiki-link\" data-key=\"86a503652ed5cc9d8e2b0252a480b5e1\">encryption<\/a> (IBE) has so far proven to be one of the best security solutions to protect eHealth record data.<sup id=\"rdp-ebb-cite_ref-QadirIEEE17_22-0\" class=\"reference\"><a href=\"#cite_note-QadirIEEE17-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-YehASecure16_23-0\" class=\"reference\"><a href=\"#cite_note-YehASecure16-23\">[23]<\/a><\/sup> The encryption algorithm deals with problems found in common cryptographic techniques using any thread as a public key. The system can further enhance the security of health records by adding authentication procedures to three connected servers. In this system, communication between three servers uses IBE to encrypt data, such that each server can perform the encryption and decryption process during the data exchange. Only servers with authorized IDs can access and extract health record data. Currently, test results show performance relative to the speed of the algorithm used in the system.<sup id=\"rdp-ebb-cite_ref-IslamTheInter15_24-0\" class=\"reference\"><a href=\"#cite_note-IslamTheInter15-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TahirASystem20_25-0\" class=\"reference\"><a href=\"#cite_note-TahirASystem20-25\">[25]<\/a><\/sup>\n<\/p><p>As discussed here, and by Vilela <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-VilelaLooking20_26-0\" class=\"reference\"><a href=\"#cite_note-VilelaLooking20-26\">[26]<\/a><\/sup>, cloud storage involves sending data over the internet to a remote cloud data center for storage. Cloud storage is delivered according to demand, capacity, and cost requirements, giving the customer control over how they invest in and manage their data storage infrastructure. This model has the advantage of lending speed, scale, and durability to an organization's eHealth data management.\n<\/p><p>Running eHealth systems in the cloud has additional advantages as well, including:\n<\/p>\n<ul><li> Ease of access using a web browser and integrated single sign-on (SSO)<\/li>\n<li> No requirement for a virtual private network (VPN) to access other sites or networks<\/li>\n<li> Simplified data management and on-demand scalability<\/li>\n<li> No overhead cost to maintain the physical infrastructure<\/li>\n<li> No hardware-related post-warranty charges for the physical infrastructure<\/li>\n<li> Little to no additional power consumption<\/li><\/ul>\n<p>Mechanism commonly used in the cloud also lend to eHealth systems. For example, one of the major schemes in healthcare systems is attribute-based encryption for data. Encryption provides high-class access control for every user, as well as revocation, scalability, dynamic user management, and traceability. Users have high-grade login access, which can be integrated with two-factor authentication (2FA) as well as one-time passwords (OTP) for every access. Similarly, users\u2019 access can be revoked from a centralized management console at any point in time by the administrator. The most important advantage is that every access can be recorded and traced, and clipping can be implemented for each session.<sup id=\"rdp-ebb-cite_ref-HaquePrivacy20_27-0\" class=\"reference\"><a href=\"#cite_note-HaquePrivacy20-27\">[27]<\/a><\/sup> Privileged access management (PAM) is another technology which can be used if there are many subsystems that a user needs to access at a single point in time.\n<\/p><p>Another cloud-based scheme for eHealth systems involves robust and secure access control, which resolves single-point performance bottleneck problems. This solves the majority of security access control issues, as the certificate needs to be trusted and issued from a certified hosting platform. The main disadvantage, however, is that there is no process or mechanism for attribute revocation.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Security_issues_of_cloud-based_eHealth_systems\">Security issues of cloud-based eHealth systems<\/span><\/h2>\n<p>Despite the advantages cloud computing brings to eHealth, it still has its own security challenges which emerge from time to time due to inadequate compliance with proper security protocols and best practices. Some of those challenges include<sup id=\"rdp-ebb-cite_ref-StamatellisAPriv20_28-0\" class=\"reference\"><a href=\"#cite_note-StamatellisAPriv20-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-YaqoobSecurity19_29-0\" class=\"reference\"><a href=\"#cite_note-YaqoobSecurity19-29\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SehHealth20_30-0\" class=\"reference\"><a href=\"#cite_note-SehHealth20-30\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiuEnhanc20_31-0\" class=\"reference\"><a href=\"#cite_note-LiuEnhanc20-31\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DangASurv19_32-0\" class=\"reference\"><a href=\"#cite_note-DangASurv19-32\">[32]<\/a><\/sup>:\n<\/p>\n<ul><li> <b>Confidentiality<\/b>: Confidentiality is a process or mechanism of safeguarding patient health data from unauthorized access from public or internal users. Unauthorized access is dangerous and can potentially result in data leakage and can even cause serious damage to businesses. As the number of devices housing patient data increases, so does the potential threat (i.e., attack surface area) of exposing that data to external parties. Confidentiality is important in the healthcare industry, as the patient can be reluctant to give personal details to doctors if they are not confident the doctor's office will be able to keep that information confidential. By implementing access control and using encryption techniques, confidentiality can be achieved.<\/li><\/ul>\n<ul><li> <b><a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">Data integrity<\/a><\/b>: Data integrity is required to ensure that data are not maliciously changed at any single point in time. This is especially important given that the <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Insurance_Portability_and_Accountability_Act\" title=\"Health Insurance Portability and Accountability Act\" class=\"wiki-link\" data-key=\"b70673a0117c21576016cb7498867153\">Health Insurance Portability and Accountability Act<\/a> (HIPAA) rules state that covered entities must implement procedures and policies to protect electronic healthcare information from improper destruction or alteration. Data integrity can be achieved by using a hashing mechanism or checksum for all data. This is most effectively done by implementing <a href=\"https:\/\/www.limswiki.org\/index.php\/Blockchain\" title=\"Blockchain\" class=\"wiki-link\" data-key=\"ae8b186c311716aca561aaee91944f8e\">blockchain<\/a> technology, as it is nearly impossible to change the hash of data with blockchain; it will change the entire chain if any of the hashes are changed.<\/li><\/ul>\n<ul><li> <b>Data availability<\/b>: Data availability says that information must be available at all times. Business-critical systems should be clustered or must have high availability to guarantee maximum uptime without service interruptions.<\/li><\/ul>\n<ul><li> <b>Data access violations and breaches<\/b>: These can negatively impact a business' public image and the the trust customers and partners have in the business. Intellectual property and sensitive data lost to these incidents can lead to other financial costs related to data forensics, communication to data owners, and more.<\/li><\/ul>\n<ul><li> <b>Inappropriate updates and fixes<\/b>: As cloud computing is a shared resource, any misconfiguration of the datacenter\u2014a common cloud challenge\u2014will potentially lead to compromising the customer data hosted within that datacenter.<\/li><\/ul>\n<ul><li> <b>Lack of security technologies<\/b>: The biggest challenge during the transition to cloud computing is the implementation of an appropriate security architecture to withstand cyberattacks. Unfortunately, this process remains a mystery for several organizations. Data are exposed to different threats when organizations assume that cloud migration is a \u201clift-and-shift\u201d endeavor of simply porting their existing IT stack and security controls to a cloud environment. A lack of knowledge of the shared security responsibility is also a contributing factor.<\/li><\/ul>\n<ul><li> <b>Account hijacking<\/b>: Whether by \"phishing,\" cloud system exploit, or direct attacks on a system, an internal user account can become compromised by an external actor, giving them access to sensitive information.<\/li><\/ul>\n<ul><li> <b>Insider threat<\/b>: In-house personnel can also perform malicious attacks on company assets by hacking servers or accessing and saving sensitive patient data, company emails, and more from insufficiently secured devices and programs.<\/li><\/ul>\n<ul><li> <b>Unsecured APIs<\/b>: Cloud computing providers develop a range of user software and APIs to allow customers to manage and interact with cloud services. The security and availability of standard cloud services are linked to the security of those APIs. Poorly designed APIs can lead to misuse or theft of information. Exposed, broken, and hacked APIs inevitably lead to realized data breaches.<\/li><\/ul>\n<p>These challenges and others highlight the need for healthcare systems to understand the security and regulatory requirements necessary for keeping data and information safe in the cloud.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Classification_of_security_solutions_in_eHealth_systems\">Classification of security solutions in eHealth systems<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Cryptographic_security\">Cryptographic security<\/span><\/h3>\n<p>The continued advancement of information technology and data communications strengthens the exchange of highly sensitive medical information. Electronic health systems are widely used, and many medical facilities rely on the transmission and receipt of medical information online and across local networks. Over the years, many security systems have been introduced to monitor patient privacy and ensure the safety of interchangeable medical data. Cryptography is one of the primary techniques that provides security for eHealth systems.\n<\/p><p>The cryptographic method allows various computations to take place directly from encrypted data, without the need to define them. As such, encryption schemes with homomorphic structures can be useful in building confidentiality agreements, where confidential information remains protected not only during exchange and storage, but also processing.\n<\/p><p>Cryptography is a method of compiling and validating data, and then securely exchanging them, with the goal of keeping track of those data set up for them to investigate. Countless methods have been suggested to protect patient health care data. However, cryptographic methods can be seen in two forms: symmetric-key cryptography and asymmetric-key cryptography, where the former uses the same encryption and coding key while the latter uses different keys. These methods heavily rely on mechanisms such as public key encryption (PKE), symmetric key encryption (SKE), and a few native cryptographic devices used to protect the eHealth cloud. In the case of PKE, an open key or a pair of encryption and spelling keys are used; however, SKE-based methods use an equally shared private key as clearly defined in the PKE and SKE cryptography methods explained by Huang <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-HuangSecure18_33-0\" class=\"reference\"><a href=\"#cite_note-HuangSecure18-33\">[33]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"PKE.2C_SKE.2C_and_other_encryption_methods\">PKE, SKE, and other encryption methods<\/span><\/h4>\n<p>In PKE schemes, two different key sets are considered: public key and secret key. Pairs of private and public data encryption and encryption keys are use with PKE, whereas SKE-based approaches use the same shared private key. Other cryptographic schemes include several encryption schemes such as attribute-based encryption (ABE), searchable encryption (SE), proxy encryption, homomorphic encryption, and identity-based encryption (IBE).\n<\/p><p>Encryption is vital to data security. Encryption locks down electronic information so that no one other than the holder of the key can access the information. Encryption even works if the organizer breaks the organization\u2019s security control on the information. The process involves encrypting data in one place, then transferring data over the network, and then deciphering data in the cloud, preventing data from being stolen during transfer.\n<\/p><p>Cryptography and encryption have had a profound effect on the security of eHealth systems.<sup id=\"rdp-ebb-cite_ref-CelestiHowTo19_34-0\" class=\"reference\"><a href=\"#cite_note-CelestiHowTo19-34\">[34]<\/a><\/sup> There are some non-cryptographic methods that can also provide security, but these are not widely used as they provide partial security for the eHealth cloud under the security provided by crypto methods. Therefore, these systems are used with crypto methods in hybrid cloud systems. Other examples of encryption mechanisms include:\n<\/p>\n<ul><li> Broadcast encryption programs: This type of \"stream\" encryption allows a mid-stream broadcast website to securerly transmit data to an associate while minimizing the high-speed communications required.<sup id=\"rdp-ebb-cite_ref-SchizaProposal19_35-0\" class=\"reference\"><a href=\"#cite_note-SchizaProposal19-35\">[35]<\/a><\/sup><\/li><\/ul>\n<ul><li> Qualified encryption: Privacy-based encryption can also be a form of public key cryptography in which the user\u2019s private key and the security unit deletion of text definitions is only possible if the set of user key symbols matches the text symbols.<sup id=\"rdp-ebb-cite_ref-LinAMulti21_36-0\" class=\"reference\"><a href=\"#cite_note-LinAMulti21-36\">[36]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MehmoodAuth19_37-0\" class=\"reference\"><a href=\"#cite_note-MehmoodAuth19-37\">[37]<\/a><\/sup><\/li><\/ul>\n<ul><li> Blockchain-based encryption: In the eHealth system, the verification, preservation, and synchronization of EMRs have always been a serious problem, and the random distribution of patient records will present various risks to the patient\u2019s privacy. Placing hash counts in electronic medical data and keeping the corresponding value in the blockchain ensure its consistency and integrity. Khezr <i>et al.<\/i> demonstrate this using blockchain and Ethereum.<sup id=\"rdp-ebb-cite_ref-KhezrBlock19_38-0\" class=\"reference\"><a href=\"#cite_note-KhezrBlock19-38\">[38]<\/a><\/sup> In their work, details listed in clinical records were removed from the Ethereum blockchain, and a good agreement was submitted to the Ethereum blockchain to accept the viewing of word searches instead of the third-party person.<sup id=\"rdp-ebb-cite_ref-KhezrBlock19_38-1\" class=\"reference\"><a href=\"#cite_note-KhezrBlock19-38\">[38]<\/a><\/sup><\/li><\/ul>\n<ul><li> Searchable symmetric encryption (SSE): SSE allows users transferring data to the cloud to query the data entered without the risk of data loss. Although interest in SSE is growing, current research does not examine how structures under SSE are constructed and how they end up in various areas of the SSE system. Most applications use a file table, where the appropriate file size and sublinear search can be done using the inverted file list. Direct referrals can only be obtained using the direct reference.<sup id=\"rdp-ebb-cite_ref-LiAchiev17_39-0\" class=\"reference\"><a href=\"#cite_note-LiAchiev17-39\">[39]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZhangSearch17_40-0\" class=\"reference\"><a href=\"#cite_note-ZhangSearch17-40\">[40]<\/a><\/sup><\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Access_control_management\">Access control management<\/span><\/h3>\n<p>Another type of security mechanism for cloud-based eHealth is the access control management (AAM) system, where the server identifies users and determines their access rights. Authorized users then receieve tokens to access their sensitive records stored in the cloud.<sup id=\"rdp-ebb-cite_ref-EdemacuCollab20_41-0\" class=\"reference\"><a href=\"#cite_note-EdemacuCollab20-41\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TongCloud14_42-0\" class=\"reference\"><a href=\"#cite_note-TongCloud14-42\">[42]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Integrated_blockchain_model\">Integrated blockchain model<\/span><\/h3>\n<p>Kurdi <i>et al.<\/i> proposed an integrated blockchain model of a fully independent blockchain (FPB) and consortium blockchain (CB) as a means to improve the time taken for data validation.<sup id=\"rdp-ebb-cite_ref-KurdiHealthy19_43-0\" class=\"reference\"><a href=\"#cite_note-KurdiHealthy19-43\">[43]<\/a><\/sup> Here, FPB is used as an old database of health care facilities and CB is used to store medical data from all participating medical professionals, expanding the available options for deploying a eHealth cloud computing platform. \n<\/p><p>While the confidentiality of data converted to the cloud between institutions is not processed, medical data remain sensitive. Electronic health applications include a shared distribution of digital information between providers and patients. This transfer is an end-to-end process that must be secured and managed regularly. As hackers continue to seek new vulnerabilities to take advantage of, any cloud-driven app should be updated frequently to stay ahead of threats. However, a balance must be struck between strong security, controlled with a simple yet effective authentication method, and making authorized access reasonably simple. Additionally, the level of security and the cost associated with it must be considered. Managing security for an EMR or some other eHealth system should be a relatively painless process that doesn't increase the premium of the solution.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Custom_security_solutions\">Custom security solutions<\/span><\/h3>\n<p>Table 1 shows a variety of custom approaches to eHealth systems security, as described in the literature.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 1.<\/b> Custom approaches to eHealth systems security\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Reference\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Security and privacy\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">eHealth\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Cloud\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Contribution\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Griebel <i>et al.<\/i> 2015<sup id=\"rdp-ebb-cite_ref-GriebelAScoping15_8-1\" class=\"reference\"><a href=\"#cite_note-GriebelAScoping15-8\">[8]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Proposed secure personal eHealth system based on certificate CA to authorized users only\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ven\u010dkauskas <i>et al.<\/i> 2016<sup id=\"rdp-ebb-cite_ref-Ven.C4.8DkauskasAModel16_9-1\" class=\"reference\"><a href=\"#cite_note-Ven.C4.8DkauskasAModel16-9\">[9]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Proposed a prototype-based personalised health monitoring application\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Khan 2020<sup id=\"rdp-ebb-cite_ref-KhanAnIoT20_10-1\" class=\"reference\"><a href=\"#cite_note-KhanAnIoT20-10\">[10]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">x\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Proposed a wearable IoT-enabled heart disease prediction system using the MDCNN classifier\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yang and Hara-Azumi 2020<sup id=\"rdp-ebb-cite_ref-YangImplem20_11-1\" class=\"reference\"><a href=\"#cite_note-YangImplem20-11\">[11]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">x\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Addressed the development of IoT-based eHealth systems on both the hardware and software levels\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Chenthara <i>et al.<\/i> 2019<sup id=\"rdp-ebb-cite_ref-ChentharaSecurity19_16-1\" class=\"reference\"><a href=\"#cite_note-ChentharaSecurity19-16\">[16]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Explored efficient comprehensive security mechanisms for EHR and the techniques to maintain the integrity and confidentiality of patients\u2019 information\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Kim <i>et al.<\/i> 2019<sup id=\"rdp-ebb-cite_ref-KimMPPDS19_18-1\" class=\"reference\"><a href=\"#cite_note-KimMPPDS19-18\">[18]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">x\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Proposed a novel collaborative eHealth system, which supports multi-level privacy-preserving data sharing (MPPDS)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Zhang <i>et al.<\/i> 2018<sup id=\"rdp-ebb-cite_ref-ZhangHealth18_44-0\" class=\"reference\"><a href=\"#cite_note-ZhangHealth18-44\">[44]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Proposed HealthDep, the first secure and efficient encrypted EMR deduplication scheme for cloud-assisted eHealth systems\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yeh <i>et al.<\/i> 2018<sup id=\"rdp-ebb-cite_ref-YehCloud18_45-0\" class=\"reference\"><a href=\"#cite_note-YehCloud18-45\">[45]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Proposed a benchmark lossless method which is suitable for remoting health monitoring systems\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Avila <i>et al.<\/i> 2017<sup id=\"rdp-ebb-cite_ref-AvilaAppli17_46-0\" class=\"reference\"><a href=\"#cite_note-AvilaAppli17-46\">[46]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reviewed the applications developed in the health industry which are focused on patient care from home and implemented a service-oriented architecture (SOA) design\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Guo <i>et al.<\/i> 2014<sup id=\"rdp-ebb-cite_ref-GuoAPrivacy14_47-0\" class=\"reference\"><a href=\"#cite_note-GuoAPrivacy14-47\">[47]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Provided an answer for privacy-preserving knowledge storage by group action of PRF-based key management for unlikability, a research and access pattern concealment theme supported redundancy, and a secure assortment technique for privacy conserving keyword search\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Liu <i>et al.<\/i> 2018<sup id=\"rdp-ebb-cite_ref-LiuSecure18_48-0\" class=\"reference\"><a href=\"#cite_note-LiuSecure18-48\">[48]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2713\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Proposed automatic internet of things (IoT) building blocks to exchange the information between devices and integrated cloud computing\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Proposed_solutions_for_security_in_cloud-based_eHealth_systems\">Proposed solutions for security in cloud-based eHealth systems<\/span><\/h2>\n<p>The conventional approach to healthcare practices in the world has been changed by the growth of information and communication technologies. In the partial abandonment of paper-based medical procedures for more electronic procedures, this development is well noticed, particularly in most developed countries of the world. The need to federate and integrate electronic health information from medical research <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratories<\/a>, hospitals, and health insurance companies has led to the development of a concept called electronic health or eHealth.<sup id=\"rdp-ebb-cite_ref-EgendiImprov18_49-0\" class=\"reference\"><a href=\"#cite_note-EgendiImprov18-49\">[49]<\/a><\/sup> Simply put, eHealth can be defined as the use of information technology (IT) and e-commerce practices for health information processing, sharing, and management. However, these aspects can be difficult to manage in a cloud-based environment that enables the collaborative sharing of medical information across multiple administrative domains by multiple users.<sup id=\"rdp-ebb-cite_ref-LiagkouAttack19_50-0\" class=\"reference\"><a href=\"#cite_note-LiagkouAttack19-50\">[50]<\/a><\/sup> \n<\/p><p>To be sure, cloud computing has many benefits, including the timely and seamless transfer and sharing of medical information across geophysical boundaries. Cloud computing also simplifies the management of infrastructure for healthcare providers and has provided them with ample opportunity to familiarize themselves with IT and cloud service providers. Various academic papers have established cloud computing as having numerous advantages, including scalability, cost-effectiveness, greater agility, and improved collaborative resource sharing.<sup id=\"rdp-ebb-cite_ref-AbuKhousaEHealth12_51-0\" class=\"reference\"><a href=\"#cite_note-AbuKhousaEHealth12-51\">[51]<\/a><\/sup>\n<\/p><p>However, despite the various advantages, there are also security and privacy challenges that urgently deserve the utmost attention for the realization of cloud computing's efficient and full-scale use.<sup id=\"rdp-ebb-cite_ref-SahmimPrivacy17_52-0\" class=\"reference\"><a href=\"#cite_note-SahmimPrivacy17-52\">[52]<\/a><\/sup> On several occasions, cryptographic and non-cryptographic approaches have been used to ensure the preservation of cloud computing security and the privacy of health information. In addition, fine-grained as well as patient-centric access control systems are frequently used to achieve electronic health privacy. This paper has reviewed these different data security measures and discussed their strengths and weaknesses. Additionally, efforts have been made to provide examples of better options for securing eHealth information.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Research_issues_and_future_directions\">Research issues and future directions<\/span><\/h2>\n<p>Going forward, several privacy and security issues related to EHRs and their shared use must be addressed. EHR data are extremely sensitive, confidential, and in particular, are stored on third-party servers, leading to significant risks in terms of data privacy and security. Some of the major concerns that require further action include:\n<\/p>\n<ul><li> developing processes and procedures on how to protect and secure EHR data stored in the cloud;<\/li>\n<li> using confidential EHR and other health care data via the cloud; and<\/li>\n<li> determining the best encryption system that can be used to maintain EHR data security in the cloud.<\/li><\/ul>\n<p>The above points are not limited to EHR data and extend to other eHealth systems. This highlights the need to use secure infrastructure in eHealth systems that will ensure the privacy and security of data and thus protect patient privacy and trust. The level of privacy required is high, making privacy considerations one of the most important aspects of the healthcare industry. Maintaining confidentiality through the sharing and used of medical records is essential to detecting and preventing fraud. Keeping track of data sources and programs is advisable.\n<\/p><p>The recommendations behind this research can be further enhanced by verifying user authenticity and privacy in cloud-based big data. A secure authentication method that incorporates tree-based hashing in the authorization structure, used for user authentication during multi-level configuration, resists fraudulent attacks and protects privacy. The authoritative structure of hierarchical attributes leads to a lot of confusion and significant problems. Artificial intelligence or big data analytics (BDA) could incorporate system design controls for each process. More recently, the usefulness of replica nodes has emerged. Leaked user data is displayed with random action keys in secure data connections. By using analytics tools, patient data can be merged into a single dashboard view, potentially improving patient outcomes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Healthcare has already adopted cloud-based solutions to solve multiple problems cost-effectively. While considering the future and growth of cloud-based healthcare services, we focused on the security of critical EHRs and other eHealth systems and proposed an identity-based secure and encrypted data-sharing technique. We have studied the existing PKE, IBE, IBBE, and ABE methods and then proposed multiple solutions to safeguard cloud-based data in eHealth systems. Appropriate security solutions should be developed and maintained to protect data security. The future of cloud-based eHealth services will be the integration of file-based and cloud-based applications that integrate a computer-based hybrid IT solution that measures the flexibility and scalability associated with cloud management and healthcare data security. This review highlights a comprehensive study of existing cloud-based eHealth solutions that incorporate cryptographic and non-cryptographic methods for protecting the privacy and security of digital data.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We thank the anonymous reviewers for their valuable comments and suggestions, which helped us to improve the content of our paper.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>R.S. and Z.A.Z.; methodology: R.S.; data curation: R.S.; resources: R.S.; writing\u2014original draft preparation: R.S.; writing\u2014review and editing: R.S. and Z.A.Z.; supervision. All authors have read and agreed to the published version of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research received no external funding.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-KruseImpact17-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KruseImpact17_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kruse, C.S.; Mileski, M.; Vijaykumar, A.G. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5640822\" target=\"_blank\">\"Impact of Electronic Health Records on Long-Term Care Facilities: Systematic Review\"<\/a>. <i>JMIR Medical Informatics<\/i> <b>5<\/b> (3): e35. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fmedinform.7958\" target=\"_blank\">10.2196\/medinform.7958<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5640822\/\" target=\"_blank\">PMC5640822<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28963091\" target=\"_blank\">28963091<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5640822\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5640822<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Impact+of+Electronic+Health+Records+on+Long-Term+Care+Facilities%3A+Systematic+Review&rft.jtitle=JMIR+Medical+Informatics&rft.aulast=Kruse%2C+C.S.%3B+Mileski%2C+M.%3B+Vijaykumar%2C+A.G.+et+al.&rft.au=Kruse%2C+C.S.%3B+Mileski%2C+M.%3B+Vijaykumar%2C+A.G.+et+al.&rft.date=2017&rft.volume=5&rft.issue=3&rft.pages=e35&rft_id=info:doi\/10.2196%2Fmedinform.7958&rft_id=info:pmc\/PMC5640822&rft_id=info:pmid\/28963091&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5640822&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ButphengSecur20-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ButphengSecur20_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Butpheng, C.; Yeh, K.-H.; Xiong, H. (2020). \"Security and Privacy in IoT-Cloud-Based e-Health Systems\u2014A Comprehensive Review\". <i>Symmetry<\/i> <b>12<\/b> (7): 1191. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fsym12071191\" target=\"_blank\">10.3390\/sym12071191<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Security+and+Privacy+in+IoT-Cloud-Based+e-Health+Systems%E2%80%94A+Comprehensive+Review&rft.jtitle=Symmetry&rft.aulast=Butpheng%2C+C.%3B+Yeh%2C+K.-H.%3B+Xiong%2C+H.&rft.au=Butpheng%2C+C.%3B+Yeh%2C+K.-H.%3B+Xiong%2C+H.&rft.date=2020&rft.volume=12&rft.issue=7&rft.pages=1191&rft_id=info:doi\/10.3390%2Fsym12071191&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IsmailBlock20-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IsmailBlock20_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ismail, L.; Materwala, H. (2020). \"Blockchain Paradigm for Healthcare: Performance Evaluation\". <i>Symmetry<\/i> <b>12<\/b> (8): 1200. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fsym12081200\" target=\"_blank\">10.3390\/sym12081200<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Blockchain+Paradigm+for+Healthcare%3A+Performance+Evaluation&rft.jtitle=Symmetry&rft.aulast=Ismail%2C+L.%3B+Materwala%2C+H.&rft.au=Ismail%2C+L.%3B+Materwala%2C+H.&rft.date=2020&rft.volume=12&rft.issue=8&rft.pages=1200&rft_id=info:doi\/10.3390%2Fsym12081200&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MalluhiDecent20-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MalluhiDecent20_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Malluhi, Q.; Tran, V.D.; Trinh, V.C (2020). \"Decentralized Broadcast Encryption Schemes with Constant Size Ciphertext and Fast Decryption\". <i>Symmetry<\/i> <b>12<\/b> (6): 969. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fsym12060969\" target=\"_blank\">10.3390\/sym12060969<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Decentralized+Broadcast+Encryption+Schemes+with+Constant+Size+Ciphertext+and+Fast+Decryption&rft.jtitle=Symmetry&rft.aulast=Malluhi%2C+Q.%3B+Tran%2C+V.D.%3B+Trinh%2C+V.C&rft.au=Malluhi%2C+Q.%3B+Tran%2C+V.D.%3B+Trinh%2C+V.C&rft.date=2020&rft.volume=12&rft.issue=6&rft.pages=969&rft_id=info:doi\/10.3390%2Fsym12060969&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HassenTowards20-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HassenTowards20_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hassen, O.A.; Abdulhissein, A.A.; Darwish, S.M. et al. (2020). \"Towards a Secure Signature Scheme Based on Multimodal Biometric Technology: Application for IOT Blockchain Network\". <i>Symmetry<\/i> <b>12<\/b> (10): 1699. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fsym12101699\" target=\"_blank\">10.3390\/sym12101699<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+a+Secure+Signature+Scheme+Based+on+Multimodal+Biometric+Technology%3A+Application+for+IOT+Blockchain+Network&rft.jtitle=Symmetry&rft.aulast=Hassen%2C+O.A.%3B+Abdulhissein%2C+A.A.%3B+Darwish%2C+S.M.+et+al.&rft.au=Hassen%2C+O.A.%3B+Abdulhissein%2C+A.A.%3B+Darwish%2C+S.M.+et+al.&rft.date=2020&rft.volume=12&rft.issue=10&rft.pages=1699&rft_id=info:doi\/10.3390%2Fsym12101699&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbdulghaniAStudy19-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbdulghaniAStudy19_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abdulghani, H.A.; Nijdam, N.A.; Collen, A. et al. (2019). \"A Study on Security and Privacy Guidelines, Countermeasures, Threats: IoT Data at Rest Perspective\". <i>Symmetry<\/i> <b>11<\/b> (6): 774. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fsym11060774\" target=\"_blank\">10.3390\/sym11060774<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Study+on+Security+and+Privacy+Guidelines%2C+Countermeasures%2C+Threats%3A+IoT+Data+at+Rest+Perspective&rft.jtitle=Symmetry&rft.aulast=Abdulghani%2C+H.A.%3B+Nijdam%2C+N.A.%3B+Collen%2C+A.+et+al.&rft.au=Abdulghani%2C+H.A.%3B+Nijdam%2C+N.A.%3B+Collen%2C+A.+et+al.&rft.date=2019&rft.volume=11&rft.issue=6&rft.pages=774&rft_id=info:doi\/10.3390%2Fsym11060774&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KangTheDesign16-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KangTheDesign16_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kang, J.; Chung, H.; Lee, J. et al. (2016). \"The Design and Analysis of a Secure Personal Healthcare System Based on Certificates\". <i>Symmetry<\/i> <b>8<\/b> (11): 129. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fsym8110129\" target=\"_blank\">10.3390\/sym8110129<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Design+and+Analysis+of+a+Secure+Personal+Healthcare+System+Based+on+Certificates&rft.jtitle=Symmetry&rft.aulast=Kang%2C+J.%3B+Chung%2C+H.%3B+Lee%2C+J.+et+al.&rft.au=Kang%2C+J.%3B+Chung%2C+H.%3B+Lee%2C+J.+et+al.&rft.date=2016&rft.volume=8&rft.issue=11&rft.pages=129&rft_id=info:doi\/10.3390%2Fsym8110129&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GriebelAScoping15-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GriebelAScoping15_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-GriebelAScoping15_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Griebel, L.; Prokosch, H.-U.; K\u00f6pcke, F. et al. (2015). \"A scoping review of cloud computing in healthcare\". <i>BMC Medical Informatics and Decision Making<\/i> <b>15<\/b>: 17. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12911-015-0145-7\" target=\"_blank\">10.1186\/s12911-015-0145-7<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+scoping+review+of+cloud+computing+in+healthcare&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Griebel%2C+L.%3B+Prokosch%2C+H.-U.%3B+K%C3%B6pcke%2C+F.+et+al.&rft.au=Griebel%2C+L.%3B+Prokosch%2C+H.-U.%3B+K%C3%B6pcke%2C+F.+et+al.&rft.date=2015&rft.volume=15&rft.pages=17&rft_id=info:doi\/10.1186%2Fs12911-015-0145-7&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Ven.C4.8DkauskasAModel16-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Ven.C4.8DkauskasAModel16_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-Ven.C4.8DkauskasAModel16_9-1\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ven\u010dkauskas, A.; \u0160tuikys, V.; Toldinas, J. et al. (2016). \"A Model-Driven Framework to Develop Personalized Health Monitoring\". <i>Symmetry<\/i> <b>8<\/b> (7): 65. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fsym8070065\" target=\"_blank\">10.3390\/sym8070065<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Model-Driven+Framework+to+Develop+Personalized+Health+Monitoring&rft.jtitle=Symmetry&rft.aulast=Ven%C4%8Dkauskas%2C+A.%3B+%C5%A0tuikys%2C+V.%3B+Toldinas%2C+J.+et+al.&rft.au=Ven%C4%8Dkauskas%2C+A.%3B+%C5%A0tuikys%2C+V.%3B+Toldinas%2C+J.+et+al.&rft.date=2016&rft.volume=8&rft.issue=7&rft.pages=65&rft_id=info:doi\/10.3390%2Fsym8070065&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhanAnIoT20-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KhanAnIoT20_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-KhanAnIoT20_10-1\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khan, M.A. (2020). \"An IoT Framework for Heart Disease Prediction Based on MDCNN Classifier\". <i>IEEE Access<\/i> <b>8<\/b>: 34717\u201327. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2020.2974687\" target=\"_blank\">10.1109\/ACCESS.2020.2974687<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+IoT+Framework+for+Heart+Disease+Prediction+Based+on+MDCNN+Classifier&rft.jtitle=IEEE+Access&rft.aulast=Khan%2C+M.A.&rft.au=Khan%2C+M.A.&rft.date=2020&rft.volume=8&rft.pages=34717%E2%80%9327&rft_id=info:doi\/10.1109%2FACCESS.2020.2974687&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YangImplem20-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-YangImplem20_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-YangImplem20_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yang, M.; Hara-Azumi, Y. (2020). \"Implementation of Lightweight eHealth Applications on a Low-Power Embedded Processor\". <i>IEEE Access<\/i> <b>8<\/b>: 121724\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2020.3006901\" target=\"_blank\">10.1109\/ACCESS.2020.3006901<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementation+of+Lightweight+eHealth+Applications+on+a+Low-Power+Embedded+Processor&rft.jtitle=IEEE+Access&rft.aulast=Yang%2C+M.%3B+Hara-Azumi%2C+Y.&rft.au=Yang%2C+M.%3B+Hara-Azumi%2C+Y.&rft.date=2020&rft.volume=8&rft.pages=121724%E2%80%9332&rft_id=info:doi\/10.1109%2FACCESS.2020.3006901&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuoADecrypt20-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuoADecrypt20_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Guo, L.; Li, Z.; Yau, W.-C. et al. (2020). \"A Decryptable Attribute-Based Keyword Search Scheme on eHealth Cloud in Internet of Things Platforms\". <i>IEEE Access<\/i> <b>8<\/b>: 26107\u201318. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2020.2971088\" target=\"_blank\">10.1109\/ACCESS.2020.2971088<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Decryptable+Attribute-Based+Keyword+Search+Scheme+on+eHealth+Cloud+in+Internet+of+Things+Platforms&rft.jtitle=IEEE+Access&rft.aulast=Guo%2C+L.%3B+Li%2C+Z.%3B+Yau%2C+W.-C.+et+al.&rft.au=Guo%2C+L.%3B+Li%2C+Z.%3B+Yau%2C+W.-C.+et+al.&rft.date=2020&rft.volume=8&rft.pages=26107%E2%80%9318&rft_id=info:doi\/10.1109%2FACCESS.2020.2971088&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EdemacuPrivacy19-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EdemacuPrivacy19_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Edemacu, K.; Park, H.K.; Jang, B. et al. (2019). \"Privacy Provision in Collaborative Ehealth With Attribute-Based Encryption: Survey, Challenges and Future Directions\". <i>IEEE Access<\/i> <b>7<\/b>: 89614\u201489636. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2925390\" target=\"_blank\">10.1109\/ACCESS.2019.2925390<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Privacy+Provision+in+Collaborative+Ehealth+With+Attribute-Based+Encryption%3A+Survey%2C+Challenges+and+Future+Directions&rft.jtitle=IEEE+Access&rft.aulast=Edemacu%2C+K.%3B+Park%2C+H.K.%3B+Jang%2C+B.+et+al.&rft.au=Edemacu%2C+K.%3B+Park%2C+H.K.%3B+Jang%2C+B.+et+al.&rft.date=2019&rft.volume=7&rft.pages=89614%E2%80%9489636&rft_id=info:doi\/10.1109%2FACCESS.2019.2925390&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaRevoc19-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaRevoc19_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ma, H.; Xie, Y.; Wang, J. et al. (2019). \"Revocable Attribute-Based Encryption Scheme With Efficient Deduplication for Ehealth Systems\". <i>IEEE Access<\/i> <b>7<\/b>: 89205\u201317. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2926627\" target=\"_blank\">10.1109\/ACCESS.2019.2926627<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Revocable+Attribute-Based+Encryption+Scheme+With+Efficient+Deduplication+for+Ehealth+Systems&rft.jtitle=IEEE+Access&rft.aulast=Ma%2C+H.%3B+Xie%2C+Y.%3B+Wang%2C+J.+et+al.&rft.au=Ma%2C+H.%3B+Xie%2C+Y.%3B+Wang%2C+J.+et+al.&rft.date=2019&rft.volume=7&rft.pages=89205%E2%80%9317&rft_id=info:doi\/10.1109%2FACCESS.2019.2926627&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CaizaReusable19-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CaizaReusable19_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-CaizaReusable19_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Caiza, J.C.; Martin, Y.-S.; Guam\u00e1n, D.S. et al. (2019). \"Reusable Elements for the Systematic Design of Privacy-Friendly Information Systems: A Mapping Study\". <i>IEEE Access<\/i> <b>7<\/b>: 66512\u201335. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2918003\" target=\"_blank\">10.1109\/ACCESS.2019.2918003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reusable+Elements+for+the+Systematic+Design+of+Privacy-Friendly+Information+Systems%3A+A+Mapping+Study&rft.jtitle=IEEE+Access&rft.aulast=Caiza%2C+J.C.%3B+Martin%2C+Y.-S.%3B+Guam%C3%A1n%2C+D.S.+et+al.&rft.au=Caiza%2C+J.C.%3B+Martin%2C+Y.-S.%3B+Guam%C3%A1n%2C+D.S.+et+al.&rft.date=2019&rft.volume=7&rft.pages=66512%E2%80%9335&rft_id=info:doi\/10.1109%2FACCESS.2019.2918003&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChentharaSecurity19-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ChentharaSecurity19_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-ChentharaSecurity19_16-1\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chenthara, S.; Ahmed, K.; Wang, H. et al. (2019). \"Security and Privacy-Preserving Challenges of e-Health Solutions in Cloud Computing\". <i>IEEE Access<\/i> <b>7<\/b>: 74361\u201382. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2919982\" target=\"_blank\">10.1109\/ACCESS.2019.2919982<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Security+and+Privacy-Preserving+Challenges+of+e-Health+Solutions+in+Cloud+Computing&rft.jtitle=IEEE+Access&rft.aulast=Chenthara%2C+S.%3B+Ahmed%2C+K.%3B+Wang%2C+H.+et+al.&rft.au=Chenthara%2C+S.%3B+Ahmed%2C+K.%3B+Wang%2C+H.+et+al.&rft.date=2019&rft.volume=7&rft.pages=74361%E2%80%9382&rft_id=info:doi\/10.1109%2FACCESS.2019.2919982&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RazaqueSurv19-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RazaqueSurv19_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Razaque, A.; Amsaad, F.; Khan, M.J. et al. (2019). \"Survey: Cybersecurity Vulnerabilities, Attacks and Solutions in the Medical Domain\". <i>IEEE Access<\/i> <b>7<\/b>: 168774\u201397. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2926627\" target=\"_blank\">10.1109\/ACCESS.2019.2926627<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Survey%3A+Cybersecurity+Vulnerabilities%2C+Attacks+and+Solutions+in+the+Medical+Domain&rft.jtitle=IEEE+Access&rft.aulast=Razaque%2C+A.%3B+Amsaad%2C+F.%3B+Khan%2C+M.J.+et+al.&rft.au=Razaque%2C+A.%3B+Amsaad%2C+F.%3B+Khan%2C+M.J.+et+al.&rft.date=2019&rft.volume=7&rft.pages=168774%E2%80%9397&rft_id=info:doi\/10.1109%2FACCESS.2019.2926627&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimMPPDS19-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KimMPPDS19_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-KimMPPDS19_18-1\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, J.W.; Edemacy, K.; Jang, B. (2019). \"MPPDS: Multilevel Privacy-Preserving Data Sharing in a Collaborative eHealth System\". <i>IEEE Access<\/i> <b>7<\/b>: 109910\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2933542\" target=\"_blank\">10.1109\/ACCESS.2019.2933542<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MPPDS%3A+Multilevel+Privacy-Preserving+Data+Sharing+in+a+Collaborative+eHealth+System&rft.jtitle=IEEE+Access&rft.aulast=Kim%2C+J.W.%3B+Edemacy%2C+K.%3B+Jang%2C+B.&rft.au=Kim%2C+J.W.%3B+Edemacy%2C+K.%3B+Jang%2C+B.&rft.date=2019&rft.volume=7&rft.pages=109910%E2%80%9323&rft_id=info:doi\/10.1109%2FACCESS.2019.2933542&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BourasDistrib20-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BourasDistrib20_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bouras, M.A.; Lu, Q.; Zhang, F.; et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7013398\" target=\"_blank\">\"Distributed Ledger Technology for eHealth Identity Privacy: State of The Art and Future Perspective\"<\/a>. <i>Sensors<\/i> <b>20<\/b> (2): 483. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs20020483\" target=\"_blank\">10.3390\/s20020483<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7013398\/\" target=\"_blank\">PMC7013398<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31952172\" target=\"_blank\">31952172<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7013398\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7013398<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Distributed+Ledger+Technology+for+eHealth+Identity+Privacy%3A+State+of+The+Art+and+Future+Perspective&rft.jtitle=Sensors&rft.aulast=Bouras%2C+M.A.%3B+Lu%2C+Q.%3B+Zhang%2C+F.%3B+et+al.&rft.au=Bouras%2C+M.A.%3B+Lu%2C+Q.%3B+Zhang%2C+F.%3B+et+al.&rft.date=2020&rft.volume=20&rft.issue=2&rft.pages=483&rft_id=info:doi\/10.3390%2Fs20020483&rft_id=info:pmc\/PMC7013398&rft_id=info:pmid\/31952172&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7013398&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SeolPrivacy18-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SeolPrivacy18_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Seol, K.; Kim, Y.-G.; Lee, E. et al. (2018). \"Privacy-Preserving Attribute-Based Access Control Model for XML-Based Electronic Health Record System\". <i>IEEE Access<\/i> <b>6<\/b>: 9114\u201428. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2018.2800288\" target=\"_blank\">10.1109\/ACCESS.2018.2800288<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Privacy-Preserving+Attribute-Based+Access+Control+Model+for+XML-Based+Electronic+Health+Record+System&rft.jtitle=IEEE+Access&rft.aulast=Seol%2C+K.%3B+Kim%2C+Y.-G.%3B+Lee%2C+E.+et+al.&rft.au=Seol%2C+K.%3B+Kim%2C+Y.-G.%3B+Lee%2C+E.+et+al.&rft.date=2018&rft.volume=6&rft.pages=9114%E2%80%9428&rft_id=info:doi\/10.1109%2FACCESS.2018.2800288&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhuAnEff18-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhuAnEff18_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhu, L.; Zhang, C.; Xu, C. et al. (2018). \"An Efficient and Privacy-Preserving Biometric Identification Scheme in Cloud Computing\". <i>IEEE Access<\/i> <b>6<\/b>: 19025\u201333. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2018.2819166\" target=\"_blank\">10.1109\/ACCESS.2018.2819166<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Efficient+and+Privacy-Preserving+Biometric+Identification+Scheme+in+Cloud+Computing&rft.jtitle=IEEE+Access&rft.aulast=Zhu%2C+L.%3B+Zhang%2C+C.%3B+Xu%2C+C.+et+al.&rft.au=Zhu%2C+L.%3B+Zhang%2C+C.%3B+Xu%2C+C.+et+al.&rft.date=2018&rft.volume=6&rft.pages=19025%E2%80%9333&rft_id=info:doi\/10.1109%2FACCESS.2018.2819166&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-QadirIEEE17-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-QadirIEEE17_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Qadir, J.; Mujeeb-U-Rahman, M.; Rehmani, M.H. et al. (2017). \"IEEE Access Special Section Editorial: Health Informatics for the Developing World\". <i>IEEE Access<\/i> <b>5<\/b>: 27818\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2017.2783118\" target=\"_blank\">10.1109\/ACCESS.2017.2783118<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=IEEE+Access+Special+Section+Editorial%3A+Health+Informatics+for+the+Developing+World&rft.jtitle=IEEE+Access&rft.aulast=Qadir%2C+J.%3B+Mujeeb-U-Rahman%2C+M.%3B+Rehmani%2C+M.H.+et+al.&rft.au=Qadir%2C+J.%3B+Mujeeb-U-Rahman%2C+M.%3B+Rehmani%2C+M.H.+et+al.&rft.date=2017&rft.volume=5&rft.pages=27818%E2%80%9323&rft_id=info:doi\/10.1109%2FACCESS.2017.2783118&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YehASecure16-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YehASecure16_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yeh, K.-H. (2016). \"A Secure IoT-Based Healthcare System With Body Sensor Networks\". <i>IEEE Access<\/i> <b>4<\/b>: 10288\u201399. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2016.2638038\" target=\"_blank\">10.1109\/ACCESS.2016.2638038<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Secure+IoT-Based+Healthcare+System+With+Body+Sensor+Networks&rft.jtitle=IEEE+Access&rft.aulast=Yeh%2C+K.-H.&rft.au=Yeh%2C+K.-H.&rft.date=2016&rft.volume=4&rft.pages=10288%E2%80%9399&rft_id=info:doi\/10.1109%2FACCESS.2016.2638038&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IslamTheInter15-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IslamTheInter15_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Islam, S.M.R.; Kwak, D.; Kabir, H. et al. (2015). \"The Internet of Things for Health Care: A Comprehensive Survey\". <i>IEEE Access<\/i> <b>3<\/b>: 678\u2013708. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2015.2437951\" target=\"_blank\">10.1109\/ACCESS.2015.2437951<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Internet+of+Things+for+Health+Care%3A+A+Comprehensive+Survey&rft.jtitle=IEEE+Access&rft.aulast=Islam%2C+S.M.R.%3B+Kwak%2C+D.%3B+Kabir%2C+H.+et+al.&rft.au=Islam%2C+S.M.R.%3B+Kwak%2C+D.%3B+Kabir%2C+H.+et+al.&rft.date=2015&rft.volume=3&rft.pages=678%E2%80%93708&rft_id=info:doi\/10.1109%2FACCESS.2015.2437951&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TahirASystem20-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TahirASystem20_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tahir, A.; Chen, F.; Khan, H.U. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7570508\" target=\"_blank\">\"A Systematic Review on Cloud Storage Mechanisms Concerning e-Healthcare Systems\"<\/a>. <i>Sensors<\/i> <b>20<\/b> (18): 5392. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs20185392\" target=\"_blank\">10.3390\/s20185392<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7570508\/\" target=\"_blank\">PMC7570508<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32967094\" target=\"_blank\">32967094<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7570508\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7570508<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Systematic+Review+on+Cloud+Storage+Mechanisms+Concerning+e-Healthcare+Systems&rft.jtitle=Sensors&rft.aulast=Tahir%2C+A.%3B+Chen%2C+F.%3B+Khan%2C+H.U.+et+al.&rft.au=Tahir%2C+A.%3B+Chen%2C+F.%3B+Khan%2C+H.U.+et+al.&rft.date=2020&rft.volume=20&rft.issue=18&rft.pages=5392&rft_id=info:doi\/10.3390%2Fs20185392&rft_id=info:pmc\/PMC7570508&rft_id=info:pmid\/32967094&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7570508&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VilelaLooking20-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VilelaLooking20_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vilela, P.H.; Rodrigues, J.J.P.C.; Righi, R.da R. et al. (2020). \"Looking at Fog Computing for E-Health through the Lens of Deployment Challenges and Applications\". <i>Sensors<\/i> <b>20<\/b> (9): 2553. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs20092553\" target=\"_blank\">10.3390\/s20092553<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Looking+at+Fog+Computing+for+E-Health+through+the+Lens+of+Deployment+Challenges+and+Applications&rft.jtitle=Sensors&rft.aulast=Vilela%2C+P.H.%3B+Rodrigues%2C+J.J.P.C.%3B+Righi%2C+R.da+R.+et+al.&rft.au=Vilela%2C+P.H.%3B+Rodrigues%2C+J.J.P.C.%3B+Righi%2C+R.da+R.+et+al.&rft.date=2020&rft.volume=20&rft.issue=9&rft.pages=2553&rft_id=info:doi\/10.3390%2Fs20092553&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HaquePrivacy20-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HaquePrivacy20_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Haque, R.U.; Hasan, A.S.M.T.; Jiang, Q. et al. (2020). \"Privacy-Preserving K-Nearest Neighbors Training over Blockchain-Based Encrypted Health Data\". <i>Electronics<\/i> <b>9<\/b> (12): 2096. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Felectronics9122096\" target=\"_blank\">10.3390\/electronics9122096<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Privacy-Preserving+K-Nearest+Neighbors+Training+over+Blockchain-Based+Encrypted+Health+Data&rft.jtitle=Electronics&rft.aulast=Haque%2C+R.U.%3B+Hasan%2C+A.S.M.T.%3B+Jiang%2C+Q.+et+al.&rft.au=Haque%2C+R.U.%3B+Hasan%2C+A.S.M.T.%3B+Jiang%2C+Q.+et+al.&rft.date=2020&rft.volume=9&rft.issue=12&rft.pages=2096&rft_id=info:doi\/10.3390%2Felectronics9122096&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StamatellisAPriv20-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StamatellisAPriv20_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stamatellis, C.; Papadopoulos, P.; Pitropakis, N. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7698751\" target=\"_blank\">\"A Privacy-Preserving Healthcare Framework Using Hyperledger Fabric\"<\/a>. <i>Sensors<\/i> <b>20<\/b> (22): 6587. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs20226587\" target=\"_blank\">10.3390\/s20226587<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7698751\/\" target=\"_blank\">PMC7698751<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/33218022\" target=\"_blank\">33218022<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7698751\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7698751<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Privacy-Preserving+Healthcare+Framework+Using+Hyperledger+Fabric&rft.jtitle=Sensors&rft.aulast=Stamatellis%2C+C.%3B+Papadopoulos%2C+P.%3B+Pitropakis%2C+N.+et+al.&rft.au=Stamatellis%2C+C.%3B+Papadopoulos%2C+P.%3B+Pitropakis%2C+N.+et+al.&rft.date=2020&rft.volume=20&rft.issue=22&rft.pages=6587&rft_id=info:doi\/10.3390%2Fs20226587&rft_id=info:pmc\/PMC7698751&rft_id=info:pmid\/33218022&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7698751&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YaqoobSecurity19-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YaqoobSecurity19_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yaqoob, T.; Abbas, H.; Atiquzzaman, M. et al. (2019). \"Security Vulnerabilities, Attacks, Countermeasures, and Regulations of Networked Medical Devices\u2014A Review\". <i>IEEE Communications Surveys & Tutorials<\/i> <b>21<\/b> (4): 3723\u201468. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCOMST.2019.2914094\" target=\"_blank\">10.1109\/COMST.2019.2914094<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Security+Vulnerabilities%2C+Attacks%2C+Countermeasures%2C+and+Regulations+of+Networked+Medical+Devices%E2%80%94A+Review&rft.jtitle=IEEE+Communications+Surveys+%26+Tutorials&rft.aulast=Yaqoob%2C+T.%3B+Abbas%2C+H.%3B+Atiquzzaman%2C+M.+et+al.&rft.au=Yaqoob%2C+T.%3B+Abbas%2C+H.%3B+Atiquzzaman%2C+M.+et+al.&rft.date=2019&rft.volume=21&rft.issue=4&rft.pages=3723%E2%80%9468&rft_id=info:doi\/10.1109%2FCOMST.2019.2914094&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SehHealth20-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SehHealth20_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Seh, A.H.; Zarour, M.; Alenezi, M. et al. (2020). \"Healthcare Data Breaches: Insights and Implications\". <i>Healthcare<\/i> <b>8<\/b> (2): 133. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fhealthcare8020133\" target=\"_blank\">10.3390\/healthcare8020133<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Healthcare+Data+Breaches%3A+Insights+and+Implications&rft.jtitle=Healthcare&rft.aulast=Seh%2C+A.H.%3B+Zarour%2C+M.%3B+Alenezi%2C+M.+et+al.&rft.au=Seh%2C+A.H.%3B+Zarour%2C+M.%3B+Alenezi%2C+M.+et+al.&rft.date=2020&rft.volume=8&rft.issue=2&rft.pages=133&rft_id=info:doi\/10.3390%2Fhealthcare8020133&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiuEnhanc20-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiuEnhanc20_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liu, H.; Crespo, R.G.; Martinez, O.S. (2020). \"Enhancing Privacy and Data Security across Healthcare Applications Using Blockchain and Distributed Ledger Concepts\". <i>Healthcare<\/i> <b>8<\/b> (3): 243. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fhealthcare8030243\" target=\"_blank\">10.3390\/healthcare8030243<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enhancing+Privacy+and+Data+Security+across+Healthcare+Applications+Using+Blockchain+and+Distributed+Ledger+Concepts&rft.jtitle=Healthcare&rft.aulast=Liu%2C+H.%3B+Crespo%2C+R.G.%3B+Martinez%2C+O.S.&rft.au=Liu%2C+H.%3B+Crespo%2C+R.G.%3B+Martinez%2C+O.S.&rft.date=2020&rft.volume=8&rft.issue=3&rft.pages=243&rft_id=info:doi\/10.3390%2Fhealthcare8030243&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DangASurv19-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DangASurv19_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dang, L.M.; Piran, M.J.; Han, D. et al. (2019). \"A Survey on Internet of Things and Cloud Computing for Healthcare\". <i>Electronics<\/i> <b>8<\/b> (7): 768. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Felectronics8070768\" target=\"_blank\">10.3390\/electronics8070768<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Survey+on+Internet+of+Things+and+Cloud+Computing+for+Healthcare&rft.jtitle=Electronics&rft.aulast=Dang%2C+L.M.%3B+Piran%2C+M.J.%3B+Han%2C+D.+et+al.&rft.au=Dang%2C+L.M.%3B+Piran%2C+M.J.%3B+Han%2C+D.+et+al.&rft.date=2019&rft.volume=8&rft.issue=7&rft.pages=768&rft_id=info:doi\/10.3390%2Felectronics8070768&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuangSecure18-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HuangSecure18_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Huang, Q.; Yue, W.; He, Y. et al. (2018). \"Secure Identity-Based Data Sharing and Profile Matching for Mobile Healthcare Social Networks in Cloud Computing\". <i>IEEE Access<\/i> <b>6<\/b>: 36584\u201394. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2018.2852784\" target=\"_blank\">10.1109\/ACCESS.2018.2852784<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Secure+Identity-Based+Data+Sharing+and+Profile+Matching+for+Mobile+Healthcare+Social+Networks+in+Cloud+Computing&rft.jtitle=IEEE+Access&rft.aulast=Huang%2C+Q.%3B+Yue%2C+W.%3B+He%2C+Y.+et+al.&rft.au=Huang%2C+Q.%3B+Yue%2C+W.%3B+He%2C+Y.+et+al.&rft.date=2018&rft.volume=6&rft.pages=36584%E2%80%9394&rft_id=info:doi\/10.1109%2FACCESS.2018.2852784&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CelestiHowTo19-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CelestiHowTo19_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Celesti, A.; Fazio, M.; M\u00e1rquez, F.G. et al. (2019). \"How to Develop IoT Cloud e-Health Systems Based on FIWARE: A Lesson Learnt\". <i>Journal of Sensor and Actuator Networks<\/i> <b>8<\/b> (1): 7. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fjsan8010007\" target=\"_blank\">10.3390\/jsan8010007<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+to+Develop+IoT+Cloud+e-Health+Systems+Based+on+FIWARE%3A+A+Lesson+Learnt&rft.jtitle=Journal+of+Sensor+and+Actuator+Networks&rft.aulast=Celesti%2C+A.%3B+Fazio%2C+M.%3B+M%C3%A1rquez%2C+F.G.+et+al.&rft.au=Celesti%2C+A.%3B+Fazio%2C+M.%3B+M%C3%A1rquez%2C+F.G.+et+al.&rft.date=2019&rft.volume=8&rft.issue=1&rft.pages=7&rft_id=info:doi\/10.3390%2Fjsan8010007&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchizaProposal19-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchizaProposal19_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schiza, E.C.; Kyprianou, T.C.; Petkov, N. et al. (2019). \"Proposal for an eHealth Based Ecosystem Serving National Healthcare\". <i>IEEE Journal of Biomedical and Health Informatics<\/i> <b>23<\/b> (3): 1346\u201357. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FJBHI.2018.2834230\" target=\"_blank\">10.1109\/JBHI.2018.2834230<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29993757\" target=\"_blank\">29993757<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Proposal+for+an+eHealth+Based+Ecosystem+Serving+National+Healthcare&rft.jtitle=IEEE+Journal+of+Biomedical+and+Health+Informatics&rft.aulast=Schiza%2C+E.C.%3B+Kyprianou%2C+T.C.%3B+Petkov%2C+N.+et+al.&rft.au=Schiza%2C+E.C.%3B+Kyprianou%2C+T.C.%3B+Petkov%2C+N.+et+al.&rft.date=2019&rft.volume=23&rft.issue=3&rft.pages=1346%E2%80%9357&rft_id=info:doi\/10.1109%2FJBHI.2018.2834230&rft_id=info:pmid\/29993757&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LinAMulti21-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LinAMulti21_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lin, H.-Y.; Jiang, Y.-R. (2021). \"A Multi-User Ciphertext Policy Attribute-Based Encryption Scheme with Keyword Search for Medical Cloud System\". <i>Applied Sciences<\/i> <b>11<\/b> (1): 63. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fapp11010063\" target=\"_blank\">10.3390\/app11010063<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Multi-User+Ciphertext+Policy+Attribute-Based+Encryption+Scheme+with+Keyword+Search+for+Medical+Cloud+System&rft.jtitle=Applied+Sciences&rft.aulast=Lin%2C+H.-Y.%3B+Jiang%2C+Y.-R.&rft.au=Lin%2C+H.-Y.%3B+Jiang%2C+Y.-R.&rft.date=2021&rft.volume=11&rft.issue=1&rft.pages=63&rft_id=info:doi\/10.3390%2Fapp11010063&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MehmoodAuth19-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MehmoodAuth19_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mehmood, Z.; Ghani, A.; Chen, G. et al. (2019). \"Authentication and Secure Key Management in E-Health Services: A Robust and Efficient Protocol Using Biometrics\". <i>IEEE Access<\/i> <b>7<\/b>: 113385\u201397. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2935313\" target=\"_blank\">10.1109\/ACCESS.2019.2935313<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Authentication+and+Secure+Key+Management+in+E-Health+Services%3A+A+Robust+and+Efficient+Protocol+Using+Biometrics&rft.jtitle=IEEE+Access&rft.aulast=Mehmood%2C+Z.%3B+Ghani%2C+A.%3B+Chen%2C+G.+et+al.&rft.au=Mehmood%2C+Z.%3B+Ghani%2C+A.%3B+Chen%2C+G.+et+al.&rft.date=2019&rft.volume=7&rft.pages=113385%E2%80%9397&rft_id=info:doi\/10.1109%2FACCESS.2019.2935313&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhezrBlock19-38\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KhezrBlock19_38-0\">38.0<\/a><\/sup> <sup><a href=\"#cite_ref-KhezrBlock19_38-1\">38.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khezr, S.; Moniruzzaman, M.; Yassine, A. et al. (2019). \"Blockchain Technology in Healthcare: A Comprehensive Review and Directions for Future Research\". <i>Applied Sciences<\/i> <b>9<\/b> (9): 1736. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fapp9091736\" target=\"_blank\">10.3390\/app9091736<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Blockchain+Technology+in+Healthcare%3A+A+Comprehensive+Review+and+Directions+for+Future+Research&rft.jtitle=Applied+Sciences&rft.aulast=Khezr%2C+S.%3B+Moniruzzaman%2C+M.%3B+Yassine%2C+A.+et+al.&rft.au=Khezr%2C+S.%3B+Moniruzzaman%2C+M.%3B+Yassine%2C+A.+et+al.&rft.date=2019&rft.volume=9&rft.issue=9&rft.pages=1736&rft_id=info:doi\/10.3390%2Fapp9091736&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiAchiev17-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiAchiev17_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H.; Yang, Y.; Dai, Y. et al. (2017). \"Achieving Secure and Efficient Dynamic Searchable Symmetric Encryption over Medical Cloud Data\". <i>IEEE Transactions on Cloud Computing<\/i> <b>8<\/b> (2): 484\u201394. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTCC.2017.2769645\" target=\"_blank\">10.1109\/TCC.2017.2769645<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Achieving+Secure+and+Efficient+Dynamic+Searchable+Symmetric+Encryption+over+Medical+Cloud+Data&rft.jtitle=IEEE+Transactions+on+Cloud+Computing&rft.aulast=Li%2C+H.%3B+Yang%2C+Y.%3B+Dai%2C+Y.+et+al.&rft.au=Li%2C+H.%3B+Yang%2C+Y.%3B+Dai%2C+Y.+et+al.&rft.date=2017&rft.volume=8&rft.issue=2&rft.pages=484%E2%80%9394&rft_id=info:doi\/10.1109%2FTCC.2017.2769645&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhangSearch17-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhangSearch17_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhang, R.; Xue, R.; Liu, L. (2017). \"Searchable Encryption for Healthcare Clouds: A Survey\". <i>IEEE Transactions on Services Computing<\/i> <b>11<\/b> (6): 978\u201396. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTSC.2017.2762296\" target=\"_blank\">10.1109\/TSC.2017.2762296<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Searchable+Encryption+for+Healthcare+Clouds%3A+A+Survey&rft.jtitle=IEEE+Transactions+on+Services+Computing&rft.aulast=Zhang%2C+R.%3B+Xue%2C+R.%3B+Liu%2C+L.&rft.au=Zhang%2C+R.%3B+Xue%2C+R.%3B+Liu%2C+L.&rft.date=2017&rft.volume=11&rft.issue=6&rft.pages=978%E2%80%9396&rft_id=info:doi\/10.1109%2FTSC.2017.2762296&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EdemacuCollab20-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EdemacuCollab20_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Edemacu, K.; Jang, B.; Kim, J.W. (2020). \"Collaborative Ehealth Privacy and Security: An Access Control With Attribute Revocation Based on OBDD Access Structure\". <i>IEEE Journal of Biomedical and Health Informatics<\/i> <b>24<\/b> (10): 2960-2972. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FJBHI.2020.2973713\" target=\"_blank\">10.1109\/JBHI.2020.2973713<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32071017\" target=\"_blank\">32071017<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Collaborative+Ehealth+Privacy+and+Security%3A+An+Access+Control+With+Attribute+Revocation+Based+on+OBDD+Access+Structure&rft.jtitle=IEEE+Journal+of+Biomedical+and+Health+Informatics&rft.aulast=Edemacu%2C+K.%3B+Jang%2C+B.%3B+Kim%2C+J.W.&rft.au=Edemacu%2C+K.%3B+Jang%2C+B.%3B+Kim%2C+J.W.&rft.date=2020&rft.volume=24&rft.issue=10&rft.pages=2960-2972&rft_id=info:doi\/10.1109%2FJBHI.2020.2973713&rft_id=info:pmid\/32071017&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TongCloud14-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TongCloud14_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tong, Y.; Sun, J.; Chow, S.S.M. et al. (2014). \"Cloud-assisted mobile-access of health data with privacy and auditability\". <i>IEEE Journal of Biomedical and Health Informatics<\/i> <b>18<\/b> (2): 419\u201329. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FJBHI.2013.2294932\" target=\"_blank\">10.1109\/JBHI.2013.2294932<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24608047\" target=\"_blank\">24608047<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cloud-assisted+mobile-access+of+health+data+with+privacy+and+auditability&rft.jtitle=IEEE+Journal+of+Biomedical+and+Health+Informatics&rft.aulast=Tong%2C+Y.%3B+Sun%2C+J.%3B+Chow%2C+S.S.M.+et+al.&rft.au=Tong%2C+Y.%3B+Sun%2C+J.%3B+Chow%2C+S.S.M.+et+al.&rft.date=2014&rft.volume=18&rft.issue=2&rft.pages=419%E2%80%9329&rft_id=info:doi\/10.1109%2FJBHI.2013.2294932&rft_id=info:pmid\/24608047&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KurdiHealthy19-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KurdiHealthy19_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kurdi, H.; Alsalmah, S.; Alatawi, A. et al. (2019). \"HealthyBroker: A Trustworthy Blockchain-Based Multi-Cloud Broker for Patient-Centered eHealth Services\". <i>Electronics<\/i> <b>8<\/b> (6): 602. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Felectronics8060602\" target=\"_blank\">10.3390\/electronics8060602<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=HealthyBroker%3A+A+Trustworthy+Blockchain-Based+Multi-Cloud+Broker+for+Patient-Centered+eHealth+Services&rft.jtitle=Electronics&rft.aulast=Kurdi%2C+H.%3B+Alsalmah%2C+S.%3B+Alatawi%2C+A.+et+al.&rft.au=Kurdi%2C+H.%3B+Alsalmah%2C+S.%3B+Alatawi%2C+A.+et+al.&rft.date=2019&rft.volume=8&rft.issue=6&rft.pages=602&rft_id=info:doi\/10.3390%2Felectronics8060602&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhangHealth18-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhangHealth18_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhang, Y.; Xu, C.; Li, H. et al. (2018). \"HealthDep: An Efficient and Secure Deduplication Scheme for Cloud-Assisted eHealth Systems\". <i>IEEE Transactions on Industrial Informatics<\/i> <b>14<\/b> (9): 4101\u201312. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTII.2018.2832251\" target=\"_blank\">10.1109\/TII.2018.2832251<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=HealthDep%3A+An+Efficient+and+Secure+Deduplication+Scheme+for+Cloud-Assisted+eHealth+Systems&rft.jtitle=IEEE+Transactions+on+Industrial+Informatics&rft.aulast=Zhang%2C+Y.%3B+Xu%2C+C.%3B+Li%2C+H.+et+al.&rft.au=Zhang%2C+Y.%3B+Xu%2C+C.%3B+Li%2C+H.+et+al.&rft.date=2018&rft.volume=14&rft.issue=9&rft.pages=4101%E2%80%9312&rft_id=info:doi\/10.1109%2FTII.2018.2832251&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YehCloud18-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YehCloud18_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yeh, L.-Y.; Chiang, P.-Y.; Tsai, Y.-L. et al. (2018). \"Cloud-Based Fine-Grained Health Information Access Control Framework for Lightweight IoT Devices with Dynamic Auditing and Attribute Revocation\". <i>IEEE Transactions on Cloud Computing<\/i> <b>6<\/b> (2): 532\u201344. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTCC.2015.2485199\" target=\"_blank\">10.1109\/TCC.2015.2485199<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cloud-Based+Fine-Grained+Health+Information+Access+Control+Framework+for+Lightweight+IoT+Devices+with+Dynamic+Auditing+and+Attribute+Revocation&rft.jtitle=IEEE+Transactions+on+Cloud+Computing&rft.aulast=Yeh%2C+L.-Y.%3B+Chiang%2C+P.-Y.%3B+Tsai%2C+Y.-L.+et+al.&rft.au=Yeh%2C+L.-Y.%3B+Chiang%2C+P.-Y.%3B+Tsai%2C+Y.-L.+et+al.&rft.date=2018&rft.volume=6&rft.issue=2&rft.pages=532%E2%80%9344&rft_id=info:doi\/10.1109%2FTCC.2015.2485199&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AvilaAppli17-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AvilaAppli17_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Avila, K.; Sanartin, P.; Jabba, D. et al. (2017). \"Applications Based on Service-Oriented Architecture (SOA) in the Field of Home Healthcare\". <i>Sensors<\/i> <b>17<\/b> (8): 1703. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs17081703\" target=\"_blank\">10.3390\/s17081703<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Applications+Based+on+Service-Oriented+Architecture+%28SOA%29+in+the+Field+of+Home+Healthcare&rft.jtitle=Sensors&rft.aulast=Avila%2C+K.%3B+Sanartin%2C+P.%3B+Jabba%2C+D.+et+al.&rft.au=Avila%2C+K.%3B+Sanartin%2C+P.%3B+Jabba%2C+D.+et+al.&rft.date=2017&rft.volume=17&rft.issue=8&rft.pages=1703&rft_id=info:doi\/10.3390%2Fs17081703&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuoAPrivacy14-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuoAPrivacy14_47-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Guo, L.; Zhang, C.; Sun, J. et al. (2014). \"A Privacy-Preserving Attribute-Based Authentication System for Mobile Health Networks\". <i>IEEE Transactions on Mobile Computing<\/i> <b>13<\/b> (9): 1927\u201341. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTMC.2013.84\" target=\"_blank\">10.1109\/TMC.2013.84<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Privacy-Preserving+Attribute-Based+Authentication+System+for+Mobile+Health+Networks&rft.jtitle=IEEE+Transactions+on+Mobile+Computing&rft.aulast=Guo%2C+L.%3B+Zhang%2C+C.%3B+Sun%2C+J.+et+al.&rft.au=Guo%2C+L.%3B+Zhang%2C+C.%3B+Sun%2C+J.+et+al.&rft.date=2014&rft.volume=13&rft.issue=9&rft.pages=1927%E2%80%9341&rft_id=info:doi\/10.1109%2FTMC.2013.84&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiuSecure18-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiuSecure18_48-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liu, Y.; Zhang, Y.; Ling, J. et al. (2018). \"Secure and fine-grained access control on e-healthcare records in mobile cloud computing\". <i>Future Generation Computer Systems<\/i> <b>78<\/b> (3): 1020\u201326. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.future.2016.12.027\" target=\"_blank\">10.1016\/j.future.2016.12.027<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Secure+and+fine-grained+access+control+on+e-healthcare+records+in+mobile+cloud+computing&rft.jtitle=Future+Generation+Computer+Systems&rft.aulast=Liu%2C+Y.%3B+Zhang%2C+Y.%3B+Ling%2C+J.+et+al.&rft.au=Liu%2C+Y.%3B+Zhang%2C+Y.%3B+Ling%2C+J.+et+al.&rft.date=2018&rft.volume=78&rft.issue=3&rft.pages=1020%E2%80%9326&rft_id=info:doi\/10.1016%2Fj.future.2016.12.027&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EgendiImprov18-49\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EgendiImprov18_49-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Elgendi, M.; Al-Ali, A.; Mohamed, A. et al. (2018). \"Improving Remote Health Monitoring: A Low-Complexity ECG Compression Approach\". <i>Diagnostics<\/i> <b>8<\/b> (1): 10. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fdiagnostics8010010\" target=\"_blank\">10.3390\/diagnostics8010010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improving+Remote+Health+Monitoring%3A+A+Low-Complexity+ECG+Compression+Approach&rft.jtitle=Diagnostics&rft.aulast=Elgendi%2C+M.%3B+Al-Ali%2C+A.%3B+Mohamed%2C+A.+et+al.&rft.au=Elgendi%2C+M.%3B+Al-Ali%2C+A.%3B+Mohamed%2C+A.+et+al.&rft.date=2018&rft.volume=8&rft.issue=1&rft.pages=10&rft_id=info:doi\/10.3390%2Fdiagnostics8010010&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiagkouAttack19-50\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiagkouAttack19_50-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liagkou, V.; Kavvadas, V.; Chronopoulos, S.K. et al. (2019). \"Attack Detection for Healthcare Monitoring Systems Using Mechanical Learning in Virtual Private Networks over Optical Transport Layer Architecture\". <i>Computation<\/i> <b>7<\/b> (2): 24. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fcomputation7020024\" target=\"_blank\">10.3390\/computation7020024<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Attack+Detection+for+Healthcare+Monitoring+Systems+Using+Mechanical+Learning+in+Virtual+Private+Networks+over+Optical+Transport+Layer+Architecture&rft.jtitle=Computation&rft.aulast=Liagkou%2C+V.%3B+Kavvadas%2C+V.%3B+Chronopoulos%2C+S.K.+et+al.&rft.au=Liagkou%2C+V.%3B+Kavvadas%2C+V.%3B+Chronopoulos%2C+S.K.+et+al.&rft.date=2019&rft.volume=7&rft.issue=2&rft.pages=24&rft_id=info:doi\/10.3390%2Fcomputation7020024&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbuKhousaEHealth12-51\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbuKhousaEHealth12_51-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">AbuKhousa, E.; Mohamed, N.; Al-Jaroodi, J. (2012). \"e-Health Cloud: Opportunities and Challenges\". <i>Future Internet<\/i> <b>4<\/b> (3): 621\u201345. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Ffi4030621\" target=\"_blank\">10.3390\/fi4030621<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=e-Health+Cloud%3A+Opportunities+and+Challenges&rft.jtitle=Future+Internet&rft.aulast=AbuKhousa%2C+E.%3B+Mohamed%2C+N.%3B+Al-Jaroodi%2C+J.&rft.au=AbuKhousa%2C+E.%3B+Mohamed%2C+N.%3B+Al-Jaroodi%2C+J.&rft.date=2012&rft.volume=4&rft.issue=3&rft.pages=621%E2%80%9345&rft_id=info:doi\/10.3390%2Ffi4030621&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SahmimPrivacy17-52\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SahmimPrivacy17_52-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sahmim, S.; Gharsellaoui, H. (2017). \"Privacy and Security in Internet-based Computing: Cloud Computing, Internet of Things, Cloud of Things: A review\". <i>Procedia Computer Science<\/i> <b>112<\/b>: 1516\u201322. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.procs.2017.08.050\" target=\"_blank\">10.1016\/j.procs.2017.08.050<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Privacy+and+Security+in+Internet-based+Computing%3A+Cloud+Computing%2C+Internet+of+Things%2C+Cloud+of+Things%3A+A+review&rft.jtitle=Procedia+Computer+Science&rft.aulast=Sahmim%2C+S.%3B+Gharsellaoui%2C+H.&rft.au=Sahmim%2C+S.%3B+Gharsellaoui%2C+H.&rft.date=2017&rft.volume=112&rft.pages=1516%E2%80%9322&rft_id=info:doi\/10.1016%2Fj.procs.2017.08.050&rfr_id=info:sid\/en.wikipedia.org:Journal:Security_and_privacy_in_cloud-based_eHealth_systems\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. However, extensive grammar, punctuation, and idea organization and clean up was required to improve the readability of the original article. In some cases important information was missing from the references, and that information was added. The original citation seven (Huh 2018) was omitted in this version as it appeared to have no connection to the topic cited. A paragraph appearing in the original introduction involving \"mists\" and service level agreements also had no bearing to the paper and appears to have been accidentally included in the original; it is omitted for this version. The original includes material from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sciencedirect.com\/science\/article\/pii\/S1110866517302797\" target=\"_blank\">Azeez and Van der Vyver 2019<\/a> in the \"Proposed solutions for security in cloud-based eHealth systems\" section; that content was omitted for this version as it appears inadvertent or plagiarized. Finally, the Discussion in the original is a complete disaster, referencing numerous articles that aren't even discussed or cited (e.g., a mention of <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/doi.org\/10.1016\/j.future.2017.02.014\" target=\"_blank\">Rahmani <i>et al.'<\/i>s UT-Gate<\/a>, as well as more almost-plagiarized content from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sciencedirect.com\/science\/article\/pii\/S1110866517302797\" target=\"_blank\">Azeez and Van der Vyver 2019<\/a>). The discussion was completely removed for this version as it too appears to be inadvertent or plagiarized content.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150611\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.880 seconds\nReal time usage: 1.467 seconds\nPreprocessor visited node count: 39141\/1000000\nPreprocessor generated node count: 35839\/1000000\nPost\u2010expand include size: 270809\/2097152 bytes\nTemplate argument size: 83872\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 769.467 1 - -total\n 89.64% 689.780 1 - Template:Reflist\n 80.50% 619.405 52 - Template:Cite_journal\n 77.38% 595.427 52 - Template:Citation\/core\n 7.13% 54.856 63 - Template:Citation\/identifier\n 4.95% 38.062 1 - Template:Infobox_journal_article\n 4.68% 36.008 1 - Template:Infobox\n 3.83% 29.447 52 - Template:Citation\/make_link\n 3.19% 24.517 80 - Template:Infobox\/row\n 3.09% 23.782 130 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12568-0!*!0!!en!5!* and timestamp 20210705150609 and revision id 42919\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Security_and_privacy_in_cloud-based_eHealth_systems\">https:\/\/www.limswiki.org\/index.php\/Journal:Security_and_privacy_in_cloud-based_eHealth_systems<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","89396bdb73ec18118ddadafe0171ee20_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/29\/Fig1_Sivan_Symmetry21_13-5.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/cd\/Fig2_Sivan_Symmetry21_13-5.png"],"89396bdb73ec18118ddadafe0171ee20_timestamp":1625497569,"2537b4210c2770719a918d28c535508d_type":"article","2537b4210c2770719a918d28c535508d_title":"Secure data outsourcing in presence of the inference problem: Issues and directions (Jebali et al. 2020)","2537b4210c2770719a918d28c535508d_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions","2537b4210c2770719a918d28c535508d_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Secure data outsourcing in presence of the inference problem: Issues and directions\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nSecure data outsourcing in presence of the inference problem: Issues and directionsJournal\n \nJournal of Information and TelecommunicationAuthor(s)\n \nJebali, Adel; Sassi, Salma; Jemai, AkderrazakAuthor affiliation(s)\n \nTunis El Manar University, Jendouba University, Carthage UniversityPrimary contact\n \nEmail: adel dot jbali at fst dot utm dot tnYear published\n \n2020Volume and issue\n \n5(1)Article #\n \n16\u201334DOI\n \n10.1080\/24751839.2020.1819633ISSN\n \n2475-1847Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.tandfonline.com\/doi\/full\/10.1080\/24751839.2020.1819633Download\n \nhttps:\/\/www.tandfonline.com\/doi\/pdf\/10.1080\/24751839.2020.1819633 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Literature review methodology \n\n3.1 Input literature \n3.2 Processing steps \n3.3 Review output \n\n\n4 Preserving confidentiality in data outsourcing scenarios \n\n4.1 Secure data outsourcing with non-communicating servers \n4.2 Secure data outsourcing: The case of communicating servers \n4.3 Discussion \n\n\n5 Data outsourcing and the inference problem \n\n5.1 Inference attacks and prevention methods \n5.2 Inference control in cloud data integration systems \n5.3 Inference control in distributed cloud database systems \n\n\n6 Proposed solution \n7 Future research directions and challenges \n8 Conclusion \n9 Acknowledgements \n\n9.1 Conflict of interest \n\n\n10 References \n11 Notes \n\n\n\nAbstract \nWith the emergence of the cloud computing paradigms, secure data outsourcing\u2014moving some or most data to a third-party provider of secure data management services\u2014has become one of the crucial challenges of modern computing. Data owners place their data among cloud service providers (CSPs) in order to increase flexibility, optimize storage, enhance data manipulation, and decrease processing time. Nevertheless, from a security point of view, access control proves to be a major concern in this situation seeing that the security policy of the data owner must be preserved when data is moved to the cloud. The lack of a comprehensive and systematic review on this topic in the available literature motivated us to review this research problem. Here, we discuss current and emerging research on privacy and confidentiality concerns in cloud-based data outsourcing and pinpoint potential issues that are still unresolved.\nKeywords: cloud computing, data outsourcing, access control, inference leakage, secrecy and privacy\n\nIntroduction \nIn light of the increasing volume and variety of data from diverse sources\u2014e.g., from health systems, social insurance systems, scientific and academic data systems, smart cities, and social networks\u2014in-house storage and processing of large collections of data has becoming very costly. Hence, modern database systems have evolved from a centralized storage architecture to a distributed one, and with it the database-as-a-service paradigm has emerged. Data owners are increasingly moving their data to cloud service providers (CSPs) in order to increase flexibility, optimize storage, enhance data manipulation, and decrease processing times. Nonetheless, security concerns are widely recognized as a major barrier to cloud computing and other data outsourcing or database-as-a-service arrangements. Users remain reluctant to place their sensitive data in the cloud due to concerns about data disclosure to potentially untrusted external parties and other malicious parties.[1] Being processed and stored externally, data owners feel they have little control over their sensitive data, consequently putting data privacy at risk. From this perspective, access control is a major challenge seeing that the security policy of a data owner must be preserved when data is moved to the cloud. Access control policies are enforced by CSPs by keeping some sensitive data separated from each other.[2] However, some techniques like encryption are helpful to better guarantee the confidentiality of data.[3][4][5] The intent of encryption is to break sensitive associations among outsourced data by encrypting some attributes of that data. However, other data security concerns exist as well. Security breaches in distributed cloud databases could be exacerbated due to inference leakage, which occurs when a malicious actor uses information from a legitimate public response to discover more sensitive information, often from metadata. During the last two decades, researchers have devoted significant effort to enforcing access control policies and privacy protection requirements externally while maintaining a balance with data utility.[6][7][8][9][10][11][12]\nIn this paper, we review the current and emerging research on privacy and confidentiality concerns in data outsourcing and highlight research directions in this field. In summary, our systematic review addresses security concerns in cloud database systems for both communicating and non-communicating servers. We also survey this research field in relation to the inference problem and the unresolved problems that are introduced. Recognizing these challenges, this paper provides an overview of our proposed (because this is an ongoing work) solution. The crux of that solution is to firstly optimize data distribution without the need to query the workload, then partition the database in the cloud by taking into consideration access control policies and data utility, before finally running a query evaluation model on a big data framework to securely process distributed queries while retaining access control. \nThe reminder of this paper is organized as follows. The next section describes the literature review methodology adopted in this paper. After that, we review emerging research on data outsourcing in the context of privacy concerns and data utility. Then we discuss data outsourcing in relation to the inference problem. Afterwards, we introduce our proposed solution to implement a secure distributed cloud database on a big data framework (Apache Spark). We close with future research directions and challenges, as well as our final conclusions.\n\nLiterature review methodology \nThe methodology for literature review adopted in this paper follows the checklist proposed by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.[13] It includes, as shown in Figure 1, three steps: input literature, processing steps, and review output.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 1. Three stages of the literature review process.\n\n\n\nInput literature \nIn this section we describe selected literature and their selection process. Firstly, our advance keyword research was conducted on the Google Scholar search engine, with a time filter from January 1, 1990 to December 31, 2019. Table 1 lists keywords used in different Google Scholar queries.\n\n\n\n\n\n\n\nTable 1. Keywords used in our literature review search.\n\n\nKeyword\n\nNumber of viewed papers\n\n\nAccess control, Data outsourcing\n\n43\n\n\nCloud computing, Authorization policies\n\n78\n\n\nDatabase, inference leakage\n\n33\n\n\nConfidentiality constraints, Cloud database\n\n41\n\n\nSecure data integration\n\n11\n\n\nBig data, Distributed query processing\n\n39\n\n\nPrivacy, data publishing\n\n24\n\n\n\nThe logical operator used between keywords during search was the \"And\" operator. Finally, from the 269 viewed papers, 43 articles were retained for review. Figure 2 shows their distribution by publication year.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 2. Distribution of the 43 articles retained for review, by publication year.\n\n\n\nProcessing steps \nDuring the review, papers were processed by identifying the problem, understanding the proposed solution process, and listing the important findings. We summarized and compared each paper with the papers associated with the similar problem. Then for each processed paper, three or four critical sentences were introduced to highlight the limits and specify potential directions that may be followed to enhance the proposed approaches. Based on our literature review, we classified the data outsourcing and access control papers into three categories, as shown in Figure 3. The first category of papers addresses the problem of secure data outsourcing when the servers in the cloud are unaware of each other. The second category addresses secure data outsourcing where interaction between servers exists and how this later can aggravate the situation. In the last category, we address data outsourcing in relation to the inference problem, as this later can exploit semantic constraints to bypass authorization policies at the cloud level.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 3. Classification scheme of the literature retained for review.\n\n\n\nReview output \nThe outcome of the methodological review process is presented later in this paper as our proposed solution. In the \"Proposed solution\" section, we present an incremental approach composed of three steps, each step treating one of the three problems mentioned in the previous subsection. We believe that our proposed solution is capable of providing good results compared to other reviewed approaches. Afterwards, we report other potential future research areas and challenges.\n\nPreserving confidentiality in data outsourcing scenarios \nThere is a consensus in the security research community about the efficiency of data outsourcing for solving data management problems.[2] This consists of moving data from in-house storage to cloud databases, while also maintaining a balance between data confidentiality and utility (Figure 4).\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 4. A representation of secure data outsourcing.\n\n\n\nCSPs are considered honest-but-curious: the database servers answer user queries correctly and do not manage stored data, but they attempt to intelligently analyze data and queries in order to learn as much information as possible from them.\nTwo powerful techniques have been proposed to enforce access control in cloud databases. The first technique exploits vertical database fragmentation to keep some sensitive data separated from each other. The second technique resorts to encryption to make a single attribute invisible to unauthorized users. These two techniques can be implemented using the following approaches:\n\n Full outsourcing: The entire in-house database is moved to the cloud. It considers vertical database fragmentation to enforce confidentiality constraints with more than two attributes by keeping them separated from each other among distributed servers. Moreover, it resorts to encryption in order to hide confidentiality constraints with a single attribute.[6]\n \"Keep a few\": This approach departs from encryption by involving the owner side. The attributes to be encrypted are stored in plain text on the owner side since this later is considered a trusted part. The rest of the database is distributed among servers while maintaining data confidentiality through vertical fragmentation.[11]\nAside from the fact that encrypting data for storing them externally carries a considerable cost[2], previous studies have primarily concentrated on non-communicating cloud servers.[6][10][11][12][14] In this situation, servers are unaware of each other and do not exchange any information. When a master node receives a query, it decomposes it and processes it locally without the need to perform a join query. In recent years, researchers have studied the effect of communication between servers on query execution, and secure query evaluation strategies have been elaborated.[4][15][16][17] \nIn the rest of this section, we discuss current and emerging research efforts in the first two of the three mentioned architectures. The third, inference control in data outsourcing, will be discussed in the following section.\n\nSecure data outsourcing with non-communicating servers \nIn 2005, Aggarwal et al. presented some of the earliest research attempting to enforce access control in database outsourcing using vertical fragmentation.[6] Under the assumption that servers do not communicate, the work aimed to split the database on two untrusted servers while preserving data privacy, with some of the attributes possibly encrypted. They demonstrated a secure fragmentation of a relation R is a triple (F1 , F2 , E) where F1 , F2 contain attributes in plain text stored in different servers and E is the set of encrypted attributes. The tuple identifier and the encrypted attributes were replicated with each fragment. The protection measures were also augmented by a query evaluation technique defining how queries on the original table can be transformed into queries on the fragmented table. \nThe work of Hudic et al.[18] introduces an approach to enforce confidentiality and privacy while outsourcing data to a CSP. The proposed technique relies on vertical fragmentation and applies only minimal encryption to preserve data exposure to malicious parties. However, the fragmentation algorithm enforces the database logic schema to be in a third normal form to produce a good fragmentation design, and the query execution cost was not proven to be minimal. \nIn 2007, Ciriani et al.[5] addressed the problem of privacy preserving data outsourcing by resorting to the combination of fragmentation and encryption. The former is exploited to break sensitive associations between attributes, while the latter enforces the privacy of singleton confidentiality constraints. The authors go on to define a formal model of minimal fragmentation and propose a heuristic minimal fragmentation algorithm to efficiently execute queries over fragments while preserving security properties. However, when a query executed over a fragment involves an attribute that is encrypted, an additional query is executed to evaluate the conditions of the attribute, leading to performance degradation by slowing down query processing.\nIn 2011, Ciriani et al.[19] addressed the concept of secure data publishing in the presence of confidentiality and visibility constraints. By modeling these two constraints as Boolean formulas and fragment as complete truth assignments, the authors rely on the Ordered Binary Decision Diagrams (OBDD) technique to check whether a fragmentation satisfies confidentiality and visibility constraints. The proposed algorithm runs using OBDD and returns a fragmentation that guarantees correctness and minimality. However, query execution cost was not investigated in this paper, and the algorithm runs only on a database schema with a single relation. \nXu et al.[1] studied the problem of finding secure fragmentation with minimum cost for query support in 2015. Firstly, they define the cost of a fragmentation F as the sum of the cost of each query Qi executed on F multiplied by the execution frequency of Qi. Secondly, they resort to using a heuristic local search graph-based approach to obtain near optimal fragmentation. The search space was modeled as a fragmentation graph, and transformation between fragmentation as a set of edges E. Then, two search strategies where proposed: a static search strategy, which is invariant with the number of steps in a solution path, as well as a dynamic search strategy based on guided local search, which guarantees the safeness of the final solution while avoiding a dead-end. However, this paper does not investigate visibility constraints, which is an important concept for data utility. Moreover, other heuristic search techniques could have been addressed (e.g., Tabu search or simulated annealing).\nThe 2009 work of Ciriani et al.[11] puts forward a new paradigm to securely publishing data in the cloud while completely departing from encryption, since encryption is sometimes considered a very rigid tool that is delicate in its configuration, while potentially slowing down query processing. The idea behind this work is to engage the owner side (assumed to be a trusted party) to store a limited portion of data (that is supposed to be encrypted) in the clear and use vertical fragmentation to break sensitive associations among data to be stored in the cloud. Their proposed algorithm computes a fragmentation solution that minimizes the load for the data owner while guaranteeing privacy concerns. Moreover, authors highlight other metrics that can be used to characterize the quality of a fragmentation and decide which attribute is affected to the client side and which attribute is externalized. However, engaging the client to enforce access control requires mediating every query in the system, which could lead to bottlenecks and negatively impact performance. \nIn 2017, Bollwein and Wiese[8] proposed a separation of duties technique based on vertical fragmentation to address the problem of preserving confidentiality when outsourcing data to a CSP. To ensure privacy requirements were met, confidentiality constraints and data dependencies were introduced. The separation of duties problem was treated as an optimization problem to maximize the utility of the fragmented database and to enhance query execution over the distributed servers. However, the optimization problem was addressed only from the point of minimizing the number of distributed servers. Additionally, when collaboration between servers is established, the separation of duties approach is no longer efficient to preserve confidentiality constraints. The NP-hardness proofness of the separation of duties problem discussed in Bollwein and Wiese[8] was later proven by the authors the following year.[9] The separation of duties problem was addressed as an optimization problem by the combination of the two famous NP-hardness problems: bin packing and vertex coloring. The bin packing problem was introduced to take into consideration the capacity constraints of the servers, with the view that fragments should be placed in a minimum number of servers without exceeding the maximum capacity. Meanwhile, vertex coloring was introduced to enforce confidentiality constraints, seeing that the association of certain attributes in the same server violates confidentiality propriety. We should note, however, that this paper studies the separation of duties problem for single-relation databases, and to make the theory applicable in practical scenarios, a many-relations database should be used.\nKeeping in mind the fact that communication between distributed servers in data outsourcing scenarios exacerbates privacy concerns, secure query evaluation strategies should be adopted. In the next subsection we investigate prior research on secure data outsourcing with communicating servers.\n\nSecure data outsourcing: The case of communicating servers \nOver the past few years, some researchers have also investigated the problem of data outsourcing with communicating servers.[4][15][16][17] Aside from attempting to guarantee confidentiality and privacy preservation when moving databases to the cloud, these works also implemented secure query evaluation strategies to retain the overall access control policy when servers communicate with each other. It is clear that when servers (containing sensitive attributes whose association is forbidden) interact through join queries, a user\u2019s privacy will be at risk. As such, secure query evaluation strategies aim to prevent the linking of sensitive attributes attempted by malicious actors.\nBuilding on previous work[4], Bkakria et al.[15] propose an approach that securely outsources data based on fragmentation and encryption. It also enforces access control when querying data by resorting to the query privacy technique. The approach examined the case of a many-relations database with new inter-table confidentiality constraints. The approach assumed that distributed servers could collude to break data confidentiality, and as such the connection between servers was intended to be based on a primary-key or foreign key. Additionally, the query evaluation model, which is based on private information retrieval, ensures sensitive attributes remain unlinkable by malicious actors using a semi-join query. However, their proposed technique enforces database schema to be normalized, and it generates a huge number of confidentiality constraints due to the transformation of both inter-table constraints to singleton and association constraints, which could affect the quality of the fragmentation algorithm. More generic queries should be considered.\nA join query integrity check was tackled in the 2016 work of di Vimercati et al.[16] Inspired by prior work[17], the authors proposed a new technique for verifying the integrity of join queries computed by potentially untrusted cloud providers. The authors also aimed to prevent servers from learning from answered queries, which could lead to a breach of users' privacy. To do so, the first showed how markers, twins, salts, and buckets can be adapted to preserve integrity when a join query is executed as a semi-join. They then introduce two strategies to minimize the size of the verification: limit the adoption of buckets and salts to twins and markers only, and represent twins and markers through slim tuples. Additionally, the authors demonstrate through their experiments how the computational and communication overhead can be limited with an integrity check.\n\nDiscussion \nTo summarize, we can classify the previously discussed approaches according to the following criterion: confidentiality constraints support, optimal distribution support, and secure query evaluation strategy support. We would like to mention that optimal distribution is treated through secure distributions that guarantee minimum query execution costs over fragments. From this point, it is clear that all mentioned approaches support access control verification through confidentiality constraints. However, query evaluations have not been tackled in all works.[8][5][11][14] Those approaches differ from the fact that some of them ensure minimum query execution costs and data utility for the database application, but other ones addressed the problem of data outsourcing with confidentiality constraints only. However, among the secure database distribution with query evaluation strategies, we find the work of Bkakria et al.[4] provides an integral framework ensuring secure database fragmentation and communication between distributed servers. Also, it shows a reasonable query execution cost.\nNevertheless, Bkakria et al. assume that the threat comes from the cloud service providers that try to collude to break sensitive association between attributes. It does not address the case of an internal threat, where a malicious user aims to bypass access control with an inference channel. As such, we present an insightful discussion about data outsourcing in the presence of the inference problem in the following section.\n\nData outsourcing and the inference problem \nAccess control models protect sensitive data from direct disclosure via direct accesses; however, they fail to prevent indirect accesses.[20] Indirect accesses via inference channels occurs when a malicious user combines the legitimate response that he received from the system with metadata (Figure 5). According to Guarnieri et al.[21], types of external information that can be combined with legitimate data in order to produce an inference channel include database schema, system semantics, statistical information, exceptions, error messages, user-defined functions, and data dependencies.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 5. An example of bypassing access control using an inference channel.\n\n\n\nAlthough access control and inference control share the same goal of preventing unauthorized data disclosure, they differ in several fundamental aspects.[22]\nTable 2 highlights the major differences between them. According to our comparison in Table 2, we find that access control is more preferable than inference control from a complexity perspective. Consequently, several researchers have attempted to replace inference control engines with access control mechanisms. We refer the interested reader to Biskup et al. (2008)[23], Biskup et al. (2010)[24], and Katos et al.[22], as the discussion of these approaches is beyond the scope of this paper.\n\n\n\n\n\n\n\nTable 2. Access control vs. inference control.\n\n\nAccess control\n\nInference control\n\n\nDirect access control\n\nIndirect access control\n\n\nDeterministic\n\nRelated to stochastic channels\n\n\nStatic, through a set of rules\n\nDynamic, varied through time and influenced by user action and queries\n\n\nNormal expense\n\nMore expensive then access control\n\n\nComputational efficiency and high accuracy of security control\n\nEfficiency and accuracy less than found with access control\n\n\nModular and capable of covering a distributed data environment\n\nAdaptability to data distribution requires complicated techniques\n\n\n\nInference attacks and prevention methods \nAccording to Farkas and Jajodia[20], there are three types of inference attacks: a statistical attack, semantic attack, and inference attack due to data mining. For each of the mentioned techniques, researchers have devoted significant efforts to dealing with the inference problem. For statistical attacks, techniques like anonymization and data perturbation have been developed to protect data from indirect access. For security threats based on data mining, techniques like privacy-preserving data mining and privacy-preserving data publishing were carried out. Furthermore, additional research has also examined semantic attacks.[25][26][27] Among the literature can be found more than one criterion to classify approaches that deal with inference. One proposed criterion is to classify these approaches according to data and schema level.[28] In such a classification, inference constraints are then classified into a schema constraints level or a data constraints level. Another criterion could classify according to the time when the inference control techniques are performed. According to this criterion, the proposed approaches are classified into two categories: design time[29][30][31][32] and query run time[25][26][33][34]\nThe purpose of inference control at \"design time\" is to detect inference channels early on and eliminate them. This approach provides better performance for the system since no monitoring module is needed when the users query the database, as a consequence improving query execution time. Nevertheless, design time approaches are too restrictive and may lead to over-classification of the data. Additionally, it requires that the designer has a firm concept of how the system will be utilized. On the other hand, \"run time\" approaches provide data availability since they monitor the suspicious queries at run time. However, run time approaches lead to performance degradation of the database server since every query needs to be checked by the inference engine. Furthermore, the inference engine needs to manage a huge number of log files and users. As a result, this could slow down query processing. In addition, run time approaches could induce a non-deterministic access control behavior (e.g., users with the same privileges may not get the same response).\nFrom this perspective, we can conclude that the main evaluation criterion of these techniques is a trade-off between availability and system performance. Some works have been elaborated to overcome these problems, especially for run time approaches. For example, Yang et al.[35] developed a new paradigm of inference control with trusted computing to push the inference control from server side to client side in order to mitigate the bottleneck on the database server. Furthermore, Staddon[36] developed a run time inference control technique that retains fast query processing. The idea behind this work was to make query processing time depend on the length of the inference channel instead of user query history.\n\nInference control in cloud data integration systems \nData outsourcing and the inference problem is an area of research that has been investigated for many years.[12][37][38][39][40][41][42][43] Inference leakage is recognized as a major barrier to cloud computing and other data outsourcing or database-as-a-service arrangements. The problem is that the designer of the system cannot anticipate the inference channels that arise on the cloud level and could lead to security breaches. \nResearchers like da Silva et al.[38] have been able to pinpoint the inference that occurs in a homogeneous peer agent through distributed data mining, calling this process a \"peer-to-peer agent-based data mining system.\" They assert that performing distributed data mining (DDM) in such extremely open distributed systems exacerbates data privacy and security issues. As a matter of fact, inference occurs in DDM when one or more peer sites learn any confidential information (e.g., a model, patterns, or data themselves) about the dataset owned by other peers during a data mining session. The authors classified inference attacks in DDM in two categories:\n\n Inside attack scenario: This scenario occurs when a peer tries to infer sensitive information from other peers in the same mining group. Depending on the number of attackers, the authors make a distinction between a single attack (when one peer behaves maliciously) and a coalition attack (when many sites collude to attack one site). Moreover, a probe attack was introduced by the authors, which is independent of the number of peers participating in the attack.\n Outside attack scenario: This scenario takes place when a set of malicious peers try to infer useful information from other peers in a different mining group. In this case, an eavesdropping channel attack is performed by malicious peers to steal information from other peers.\nAfter identifying DDM inference attacks, the authors propose an algorithm known as KDEC to control potential inside and outside attacks to particular schema for homogeneous distributed clustering. The main idea behind KDEC is to reconstruct the data from the kernel density estimates, given that a malicious peer can use the reconstruction algorithm to infer non-local data. However, the algorithm proposed by the authors needs to be improved from an accuracy point to expose further possible weakness of the KDEC schema.\nInference control in cloud integration systems has also been investigated in the last decade through the work of Haddad et al.[39], Sayah et al.[40], and Sellami et al.[41] In such systems, a mediator is defined as a unique entry point to the distributed data sources. It provides the user a unique view of the distributed data. From a security point of view, access control is a major challenge in this situation since the global policy of the mediator in the cloud level must comply with the back-end data source policies, in addition to possibly enforcing additional security properties (Figure 6). The problem is that the system (or the designer of the system) cannot anticipate the inference channels that arise due to the dependencies that appear at the mediator level.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 6. Representation of a secure cloud data integration system.\n\n\n\nThe first known work attempting to control inference in data integration systems is by Haddad et al.[39] The authors propose an incremental approach to prevent inference with functional dependencies. The proposed methodology includes three steps:\n\n Synthesize global policies: Derives the authorization rule of each virtual relation individually by the way that it preserves the local authorization of the local relations involved in the virtual relation.\n Detect violations: By resorting to a graph-based approach, this step aims at identifying all the violations that could occur using functional dependencies. Such violations are called \"transaction violations,\" consisting of a series of innocuous queries that lead to violation of authorization rules.\n Reconfigure authorization policies: The author proposes two methods to forbid the completion of each transaction violation. The first one uses a historical access control by keeping track of previous queries to evaluate the current query (this method is considered to be a run-time approach). The second one proposes to reconfigure the global authorization policies at the mediator level in a way that no authorization violation will occur (this method is considered to be at design-time of the global security policy).\nIt should be noted that Haddad et al. have discussed only semantic constraints due to functional dependencies. Neither inclusion nor multivalued dependencies were investigated. Additionally, other mapping approaches need to be discussed, such as LAV and GLAV approaches.\nInspired by Haddad et al.[39], Sellami et al.[41] propose an approach aiming to control inference in cloud integration systems. The proposed methodology resorts to formal concept analysis as a formal framework to reason about authorization rules, and functional dependencies as a source of inference. The authors adopt an access control model with authorization views and propose an incremental approach with three steps:\n\nGenerate the global policy, global schema, and global functional dependency (FD): This step takes as input a set of source schema together with their access control policies and starts by translating the schema and policies to formal contexts. Then, the global policy is generated in a way that the source rules are preserved at the global level. Next the schema of the mediator (virtual relations) is generated from the global policy to avoid useless attributes combinations (every attribute in the mediator schema is controlled by the global policy). Finally, a global FD is considered from the source FD as a formal context.\nIdentify disclosure transactions: By resorting to formal concept analysis (FCA) as a framework to reason about the global policy, the authors identify the profiles to be denied from accessing sensitive attributes at the mediator level. Then, they extract the transaction violations by reasoning about the global FD.\nReconfigure: This step is achieved by two methods. At design time, use a policy healing to complete the global policy, with additional rules to ensure that no transaction violation is achieved. At query run time, use a monitoring engine to prohibit suspicious queries.\nSayah et al.[40] have examined inference that arises in the web through a Resource Description Framework (RDF) store. They propose a fine-grained framework for RDF data, and then they exploit a closed graph to verify the consistency property of an access control policy when inference rules and authorization rules interact. Without accessing the data (at policy design-time), the authors propose an algorithm to verify if an information leakage will arise given a policy P and a set of inference rules R. Furthermore, the authors demonstrate the applicability of the access control model using a conflict resolution strategy (most specific takes precedence).\n\nInference control in distributed cloud database systems \nBiskup et al.[37] resort to using a Controlled Query Evaluation strategy (CQE) to detect inference based on the knowledge of non-confidential information contained in the outsourced fragments and prior knowledge that a malicious user might have. As CQE relies on a logic-oriented view of database systems, the main idea of this approach is to model fragmentation in a logic-oriented fashion to allow for inference proofness to be proved formally despite the semantic database constraints that an attacker may hold. This type of vertical database fragmentation technique was also considered by di Vimercati et al.[12] to ensure data confidentiality in the presence of data dependencies among attributes. Those dependencies allow unauthorized users to deduce information about sensitive attributes. In this work, the authors define three types of confidentiality violation that can be caused by data dependencies. First, this can happen when a sensitive attribute or association is exposed by the attributes in a fragment. Second, this can also occur if an attribute appearing in a fragment is also derivable from some attributes in another fragment, thus enabling linkability among such fragments. Finally, this can occur when an attribute is derivable (independently) from attributes appearing in different fragments, thus enabling linkability among these fragments. To tackle these issues, authors reformulate the problem graphically through a hyper-graph representation and then compute the closure of a fragmentation by deducing all information derivable from its fragments via dependencies to identify indirect access. Nevertheless, the major limit of this approach is that it explores the problem only in a single-relation database.\nAlthough data outsourcing was not explicitly mentioned by Turan et al.[42][43], their two works from 2017 and 2018 aim to control the inference problem caused by functional dependencies and meaningful join proactively by decomposing the relational logical schema into a set of views (to be queried by the user) where inference channels cannot appear. In 2017, the authors proposed[43] a proactive and decomposition-based inference control strategy for relational databases to prevent access to a forbidden set via direct or indirect access. The proposed decomposition algorithm controls both functional and probabilistic dependencies by breaking down those leading to the inference of a forbidden attribute set. However, this approach was considered too rigid for the fact that if the ways of associate forbidden set attributes are defined as a chain of functional dependencies, the algorithm breaks these chains from both sides for both attributes. Parting from this limit, the same researchers proposed a graph-based approach[42] consisting of proactive decomposition of the external schema in order to satisfy both the forbidden and required associations of attributes. In this work, functional dependencies are represented as a graph in which vertices are attributes and edges are functional dependencies. The inference channel is then defined as a process of searching a sub-tree in the graph containing the attributes that need to be related. Compared to their prior approach[43], this method sees the cut of the inference channel getting relaxed by cutting the chains only at a single point, consequently minimizing dependency loss. Nevertheless, like the previous technique, it leads to semantic loss and a need to query rewriting techniques to query decomposed views.\n\nProposed solution \nIn this section, we present an approach that relies on the relational model and aims to produce a set of secure sub-schemas, with each represented by a partition Pi stored exactly on one server by the CSP (Figure 7). This approach also introduces a secure distributed query evaluation strategy to efficiently request data from distributed partitions while retaining access control policies. To do that, our proposed methodology takes as input a set of functional dependencies (FD), as well as a relational schema R, and applies the following phases:\n\n Constraints generation: This phase aims at generating two types of constraints that, in addition to the confidentiality constraints, will guide the process of vertical schema partitioning. This is done through two steps: visibility constraints generation and constraint-based inference control generation. During visibility constraints generation, the constraints are enforced as soft constraints in the partitioning phase, given that their severity is less than confidentiality constraints. To generate them, we perform a semantic analysis of the relational schema in order to detect semantic relatedness between attributes and users roles. These constraints should be preserved (stay visible) when the relational schema is fragmented (i.e., we aim to maximize intra-dependency between attributes that seem to be frequently accessed by the same role while minimizing the inter-dependency between attributes in separate parts). Then during constraint-based inference control generation, the constraints are enforced (like confidentiality constraints) as hard constraints. In this step, we resort to the method proposed by Turan et al.[42] to build a functional dependencies graph and generate join chains. Then, we use a relaxed technique to cut the join chains only at a single point in order to minimize dependency loss. By cutting a join chain at a single point, the enforcement of the attributes in the left-hand side (LHS) and right-hand side (RHS) of the functional dependency can occur, with the cut point representing a confidentiality constraint. As a consequence, we guarantee that the join chain is broken.\n Schema partitioning: In this phase, we resort to hypergraph theory to represent the partitioning problem as a hypergraph constraint satisfaction problem. Then, we reformulate the problem as a multi-objective function F to be optimized. Therefore, we propose a greedy algorithm to partition the constrained hypergraph into k parts while minimizing the multi-objective function F.\n Query evaluation model: In this step, we propose a monitor module to mediate every query issued from users against data stored in distributed partitions. The monitor module is built on top of an Apache Spark system, which contains two mechanisms: a role-based access control mechanism and a history-based access control mechanism. The first mechanism checks for the user role of who issued the query and, if it is not granted to execute distributed query, then this later will be forwarded to the CSP. Otherwise, the query is forwarded to the history access control mechanism, which takes as input a set of transaction violations to be prohibited and checks if the cumulative of user past queries and the current query could complete a transaction violation. If that proves to be the case, the query is revoked.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 7. A proposed methodology to generate secure partitions and lock suspicious queries.\n\n\n\nFuture research directions and challenges \nSince the discussed works are recent, there are a number of concepts associated to access control, privacy, data outsourcing, and database semantics which could be considered to ensure better data security and utility in the cloud. As such, there are several research directions to pursue going forward.\nFunctional dependencies should be considered a threat source in data outsourcing scenarios. Unlike the approaches by Turan et al.[42][43], we aim in our future work to prevent inference from occurring in a distributed cloud database. Our graph-based approach detects inference channels caused by functional dependencies and breaks those channels by exploiting vertical database fragmentation while minimizing dependency loss.\nPrior researchers have only dealt with semantic constraints represented by functional and probabilistic dependencies as a source of inference. However, other semantic constraints\u2014e.g., inclusion dependencies, join dependencies, and multivalued dependencies\u2014should be considered as sources of inference.\nAnother future direction to examine is when the workload will become available after the database is set up in the cloud. The challenge here is in how to dynamically reallocate the distributed database fragments among distributed servers while retaining access control policy.\n\nConclusion \nIn this work, we conducted a literature review of current and emerging research on privacy and confidentiality concerns in data outsourcing. We have introduced different research efforts to ensure user privacy in the database-as-a-service paradigm with both communicating and non-communicating servers. Additionally, an insightful discussion about inference control was introduced, as well as a proposed method of tackling the issue. We also pinpoint potential issues that are still unresolved. These issues are expected to be addressed in future work.\n\nAcknowledgements \nConflict of interest \nNo potential conflict of interest was reported by the author(s).\n\nReferences \n\n\n\u2191 1.0 1.1 Xu, X.; Xiong, L.; Liu, J. (2015). \"Database Fragmentation with Confidentiality Constraints: A Graph Search Approach\". Proceedings of the 5th ACM Conference on Data and Application Security and Privacy: 263\u201370. doi:10.1145\/2699026.2699121.   \n\n\u2191 2.0 2.1 2.2 Samarati, P.; di Vimarcati, S.D.C. (2010). \"Data protection in outsourcing scenarios: Issues and directions\". Proceedings of the 5th ACM Symposium on Information, Computer and Communications Security: 1\u201314. doi:10.1145\/1755688.1755690.   \n\n\u2191 Biskup, J.; Preu\u00df, M. (2013). \"Database Fragmentation with Encryption: Under Which Semantic Constraints and A Priori Knowledge Can Two Keep a Secret?\". Data and Applications Security and Privacy XXVIII: 17\u201332. doi:10.1007\/978-3-642-39256-6_2.   \n\n\u2191 4.0 4.1 4.2 4.3 4.4 Bkakria, A.; Cuppens, F.; Cuppens-Boulahia, N. et al. (2013). \"Preserving Multi-relational Outsourced Databases Confidentiality using Fragmentation and Encryption\". JoWUA 4 (2): 39\u201362. doi:10.22667\/JOWUA.2013.06.31.039.   \n\n\u2191 5.0 5.1 5.2 Ciriani, V.; di Vimaercati, S.D.C.; Foresti, S. et al. (2007). \"Fragmentation and Encryption to Enforce Privacy in Data Storage\". Computer Security - ESORICS 2007: 171\u201386. doi:10.1007\/978-3-540-74835-9_12.   \n\n\u2191 6.0 6.1 6.2 6.3 Aggarwal, G.; Bawa, M.; Ganesan, P. et al. (2005). \"Two Can Keep a Secret: A Distributed Architecture for Secure Database Services\". Second Biennial Conference on Innovative Data Systems Research: 1\u201314. http:\/\/ilpubs.stanford.edu:8090\/659\/ .   \n\n\u2191 Alsirhani, A.; Bodorik, P. Sampalli, S. (2017). \"Improving Database Security in Cloud Computing by Fragmentation of Data\". Proceedings of the 2017 International Conference on Computer and Applications: 43\u201349. doi:10.1109\/COMAPP.2017.8079737.   \n\n\u2191 8.0 8.1 8.2 8.3 Bollwein, F.; Wiese, L. (2017). \"Separation of Duties for Multiple Relations in Cloud Databases as an Optimization Problem\". Proceedings of the 21st International Database Engineering & Applications Symposium: 98\u2013107. doi:10.1145\/3105831.3105873.   \n\n\u2191 9.0 9.1 Bollwein, F.; Wiese, L. (2018). \"On the Hardness of Separation of Duties Problems for Cloud Databases\". Proceedings of TrustBus 2018: Trust, Privacy and Security in Digital Business: 23\u201338. doi:10.1007\/978-3-319-98385-1_3.   \n\n\u2191 10.0 10.1 Ciriani, V.; di Vimercati, S.D.C.; Foresti, S. et al. (2009). \"Fragmentation Design for Efficient Query Execution over Sensitive Distributed Databases\". Proceedings of the 29th IEEE International Conference on Distributed Computing Systems: 32\u201339. doi:10.1109\/ICDCS.2009.52.   \n\n\u2191 11.0 11.1 11.2 11.3 11.4 Ciriani, V.; di Vimercati, S.D.C.; Foresti, S. et al. (2009). \"Keep a Few: Outsourcing Data While Maintaining Confidentiality\". Computing Security - ESORICS 2009: 440\u201355. doi:10.1007\/978-3-642-04444-1_27.   \n\n\u2191 12.0 12.1 12.2 12.3 di Vimercati, S.D.C.; Foresti, S.; Jajodia, S. et al. (2014). \"Fragmentation in Presence of Data Dependencies\". IEEE Transactions on Dependable and Secure Computing 11 (6): 510\u201323. doi:10.1109\/TDSC.2013.2295798.   \n\n\u2191 Moher, D.; Liberti, A.; Tetzlaff, J. et al. (2009). \"Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement\". PLoS One 6 (7): e1000097. doi:10.1371\/journal.pmed.1000097. PMC PMC2707599. PMID 19621072. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2707599 .   \n\n\u2191 14.0 14.1 di Vimercati, S.D.C.; Foresti, S.; Jajodia, S. et al. (2010). \"Fragments and loose associations: respecting privacy in data publishing\". Proceedings of the VLDB Endowment 3 (1\u20132). doi:10.14778\/1920841.1921009.   \n\n\u2191 15.0 15.1 15.2 Bkakria, A.; Cuppens, F.; Cuppens-Boulahia, N. et al. (2013). \"Confidentiality-Preserving Query Execution of Fragmented Outsourced Data\". Proceeding of ICT-EurAsia 2013: Information and Communication Technology 4 (2): 426-440. doi:10.1007\/978-3-642-36818-9_47.   \n\n\u2191 16.0 16.1 16.2 di Vimarcati, S.D.C.; Foresti, S.; Jajodia, S. et al. (2016). \"Efficient integrity checks for join queries in the cloud\". Journal of Computer Security 24 (3): 347\u201378. doi:10.3233\/JCS-160545.   \n\n\u2191 17.0 17.1 17.2 di Vimarcati, S.D.C.; Foresti, S.; Jajodia, S. et al. (2013). \"Integrity for join queries in the cloud\". IEEE Transactions on Cloud Computing 1 (2): 187\u2013200. doi:10.1109\/TCC.2013.18.   \n\n\u2191 Hudic, A.; Islam, S.; Kieseberg, P. et al. (2013). \"Data confidentiality using fragmentation in cloud computing\". International Journal of Pervasive Computing and Communications 9 (1): 37\u201351. doi:10.1108\/17427371311315743.   \n\n\u2191 Ciriani, V.; di Vimercati, S.D.C.; Foresti, S. et al. (2011). \"Enforcing Confidentiality and Data Visibility Constraints: An OBDD Approach\". Proceeding of DBSec 2011: Data and Applications Security and Privacy XXV: 44\u201359. doi:10.1007\/978-3-642-22348-8_6.   \n\n\u2191 20.0 20.1 Farkas, C.; Jajodia, S. (2002). \"The inference problem: A survey\". ACM SIGKDD Explorations Newsletter 4 (2): 6\u201311. doi:10.1145\/772862.772864.   \n\n\u2191 Guarnieri, M.; Marinovic, S.; Basin, D. (2017). \"Securing Databases from Probabilistic Inference\". Proceedings of the 2017 IEEE 30th Computer Security Foundations Symposium: 343\u2013359. doi:10.1109\/CSF.2017.30.   \n\n\u2191 22.0 22.1 Katos, V.; Vrakas, D.; Katsaros, P. (2011). \"A framework for access control with inference constraints\". Proceedings of the 2011 IEEE 35th Annual Computer Software and Applications Conference: 289\u2013297. doi:10.1109\/COMPSAC.2011.45.   \n\n\u2191 Biskup, J.; Embley, D.W.; Lochner, J.-H. (2008). \"Reducing inference control to access control for normalized database schemas\". Information Processing Letters 106 (1): 8-12. doi:10.1016\/j.ipl.2007.09.007.   \n\n\u2191 Biskup, J.; Hartmann, S.; Link, S. (2010). \"Efficient Inference Control for Open Relational Queries\". Proceedings of DBSec 2010: Data and Applications Security and Privacy XXIV: 162\u201376. doi:10.1007\/978-3-642-13739-6_11.   \n\n\u2191 25.0 25.1 Brodsky, A. (2000). \"Secure databases: Constraints, inference channels, and monitoring disclosures\". IEEE Transactions on Knowledge and Data Engineering 12 (6): 900\u201319. doi:10.1109\/69.895801.   \n\n\u2191 26.0 26.1 Chen, Y.; Chu, W.W. (2006). \"Database Security Protection Via Inference Detection\". Proceedings of ISI 2006: Intelligence and Security Informatics: 452\u201358. doi:10.1007\/11760146_40.   \n\n\u2191 Su, T.-A.; Ozsoyoglu, G. (1991). \"Controlling FD and MVD inferences in multilevel relational database systems\". IEEE Transactions on Knowledge and Data Engineering 3 (4): 474-485. doi:10.1109\/69.109108.   \n\n\u2191 Yip, R.W.; Levitt, E.N. (1998). \"Data level inference detection in database systems\". Proceedings of the 11th IEEE Computer Security Foundations Workshop: 179\u201389. doi:10.1109\/CSFW.1998.683168.   \n\n\u2191 Delugach, H.S.; Hinke, T.H. (1996). \"Wizard: A database inference analysis and detection system\". IEEE Transactions on Knowledge and Data Engineering 8 (1): 56\u201366. doi:10.1109\/69.485629.   \n\n\u2191 Hinke, T.H.; Delugach, H.S. (1992). \"AERIE: An inference modeling and detection approach for databases\" (PDF). Proceedings of IFIP WG 11.3 Sixth Working Conference on Database Security: 187\u2013201. https:\/\/apps.dtic.mil\/sti\/pdfs\/ADA298828.pdf .   \n\n\u2191 Rath, S.; Jones, D.; Hale, J. et al. (1996). \"A Tool for Inference Detection and Knowledge Discovery in Databases\". Proceedings from Database Security IX: 317\u2013332. doi:10.1007\/978-0-387-34932-9_20.   \n\n\u2191 Wang, J.; Yang, J.; Guo, F. et al. (2017). \"Resist the Database Intrusion Caused by Functional Dependency\". Proceedings of the 2017 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery: 54\u201357. doi:10.1109\/CyberC.2017.11.   \n\n\u2191 An, X.; Jutla, D.; Cercone, N. (2006). \"Dynamic inference control in privacy preference enforcement\". Proceedings of the 2006 International Conference on Privacy, Security and Trust: 1\u201310. doi:10.1145\/1501434.1501464.   \n\n\u2191 Thuraisingham, B.; Ford, W.; Collins, M. et al. (1993). \"Design and implementation of a database inference controller\". Data & Knowledge Engineering 11 (3): 271-297. doi:10.1016\/0169-023X(93)90025-K.   \n\n\u2191 Yang, Y.; Li, Y.; Deng, R.H. (2007). \"New Paradigm of Inference Control with Trusted Computing\". Proceedings of DBSec 2007: Data and Applications Security XXI: 243\u201358. doi:10.1007\/978-3-540-73538-0_18.   \n\n\u2191 Dtaddon, J. (2003). \"Dynamic inference control\". Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery: 94\u2013100. doi:10.1145\/882082.882103.   \n\n\u2191 37.0 37.1 Biskup, J.; Preu\u00df, M.; Wiese, L. (2011). \"On the Inference-Proofness of Database Fragmentation Satisfying Confidentiality Constraints\". Proceedings of ISC 2011: Information Security: 246\u201361. doi:10.1007\/978-3-642-24861-0_17.   \n\n\u2191 38.0 38.1 da Silva, J.C.; Klusch, M.; Lodi, S. et al. (2004). \"Inference attacks in peer-to-peer homogeneous distributed data mining\". Proceedings of the 16th European Conference on Artificial Intelligence: 450-54. doi:10.5555\/3000001.3000096.   \n\n\u2191 39.0 39.1 39.2 39.3 Haddad, M.; Stevovic, J.; Chiasera, A. et al. (2014). \"Access Control for Data Integration in Presence of Data Dependencies\". Proceedings of DASFAA 2014: Database Systems for Advanced Applications: 203\u201317. doi:10.1007\/978-3-319-05813-9_14.   \n\n\u2191 40.0 40.1 40.2 Sayah, T.; Coquery, E.; Thion, R. et al. (2015). \"Inference Leakage Detection for Authorization Policies over RDF Data\". Proceedings of DBSec 2015: Data and Applications Security and Privacy XXIX: 346\u201361. doi:10.1007\/978-3-319-20810-7_24.   \n\n\u2191 41.0 41.1 41.2 Sellami, M.; Hacid, M.-S.; Gammoudi, M.M. (2015). \"Inference Control in Data Integration Systems\". Proceedings of OTM 2015: On the Move to Meaningful Internet System: 285\u2013302. doi:10.1007\/978-3-319-26148-5_17.   \n\n\u2191 42.0 42.1 42.2 42.3 42.4 Turan, U.; Toroslu, I.H.; Kantarcioglu, M. (2018). \"Graph Based Proactive Secure Decomposition Algorithm for Context Dependent Attribute Based Inference Control Problem\". arXiv: 1\u201311. https:\/\/arxiv.org\/abs\/1803.00497 .   \n\n\u2191 43.0 43.1 43.2 43.3 43.4 Turan, U.; Torslu, I.H.; Kantarc\u0131o\u011flu, M. (2017). \"Secure logical schema and decomposition algorithm for proactive context dependent attribute based inference control\". Data & Knowledge Engineering 111: 1\u201321. doi:10.1016\/j.datak.2017.02.002.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. The original paper listed references alphabetically; this wiki lists them by order of appearance, by design.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\">https:\/\/www.limswiki.org\/index.php\/Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on cloud computingLIMSwiki journal articles on cybersecurityLIMSwiki journal articles on data management and sharing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 7 April 2021, at 16:40.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 584 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","2537b4210c2770719a918d28c535508d_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Secure_data_outsourcing_in_presence_of_the_inference_problem_Issues_and_directions skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Secure data outsourcing in presence of the inference problem: Issues and directions<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>With the emergence of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud computing<\/a> paradigms, secure data outsourcing\u2014moving some or most data to a third-party provider of secure data management services\u2014has become one of the crucial challenges of modern computing. Data owners place their data among cloud service providers (CSPs) in order to increase flexibility, optimize storage, enhance data manipulation, and decrease processing time. Nevertheless, from a <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">security<\/a> point of view, access control proves to be a major concern in this situation seeing that the security policy of the data owner must be preserved when data is moved to the cloud. The lack of a comprehensive and systematic review on this topic in the available literature motivated us to review this research problem. Here, we discuss current and emerging research on privacy and confidentiality concerns in cloud-based data outsourcing and pinpoint potential issues that are still unresolved.\n<\/p><p><b>Keywords<\/b>: cloud computing, data outsourcing, access control, inference leakage, secrecy and privacy\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>In light of the increasing volume and variety of data from diverse sources\u2014e.g., from health systems, social insurance systems, scientific and academic data systems, smart cities, and social networks\u2014in-house storage and processing of large collections of data has becoming very costly. Hence, modern database systems have evolved from a centralized storage architecture to a distributed one, and with it the database-as-a-service paradigm has emerged. Data owners are increasingly moving their data to cloud service providers (CSPs) in order to increase flexibility, optimize storage, enhance data manipulation, and decrease processing times. Nonetheless, <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">security<\/a> concerns are widely recognized as a major barrier to <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud computing<\/a> and other data outsourcing or database-as-a-service arrangements. Users remain reluctant to place their sensitive data in the cloud due to concerns about data disclosure to potentially untrusted external parties and other malicious parties.<sup id=\"rdp-ebb-cite_ref-XuDatabase15_1-0\" class=\"reference\"><a href=\"#cite_note-XuDatabase15-1\">[1]<\/a><\/sup> Being processed and stored externally, data owners feel they have little control over their sensitive data, consequently putting data privacy at risk. From this perspective, access control is a major challenge seeing that the security policy of a data owner must be preserved when data is moved to the cloud. Access control policies are enforced by CSPs by keeping some sensitive data separated from each other.<sup id=\"rdp-ebb-cite_ref-SamaratiData10_2-0\" class=\"reference\"><a href=\"#cite_note-SamaratiData10-2\">[2]<\/a><\/sup> However, some techniques like <a href=\"https:\/\/www.limswiki.org\/index.php\/Encryption\" title=\"Encryption\" class=\"wiki-link\" data-key=\"86a503652ed5cc9d8e2b0252a480b5e1\">encryption<\/a> are helpful to better guarantee the confidentiality of data.<sup id=\"rdp-ebb-cite_ref-BiskupData13_3-0\" class=\"reference\"><a href=\"#cite_note-BiskupData13-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BkakriaPreserv13_4-0\" class=\"reference\"><a href=\"#cite_note-BkakriaPreserv13-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CirianiFrag07_5-0\" class=\"reference\"><a href=\"#cite_note-CirianiFrag07-5\">[5]<\/a><\/sup> The intent of encryption is to break sensitive associations among outsourced data by encrypting some attributes of that data. However, other data security concerns exist as well. Security breaches in distributed cloud databases could be exacerbated due to inference leakage, which occurs when a malicious actor uses information from a legitimate public response to discover more sensitive information, often from metadata. During the last two decades, researchers have devoted significant effort to enforcing access control policies and privacy protection requirements externally while maintaining a balance with data utility.<sup id=\"rdp-ebb-cite_ref-AggarwalTwoCan05_6-0\" class=\"reference\"><a href=\"#cite_note-AggarwalTwoCan05-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlsirhaniImprov17_7-0\" class=\"reference\"><a href=\"#cite_note-AlsirhaniImprov17-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BollweinSepar17_8-0\" class=\"reference\"><a href=\"#cite_note-BollweinSepar17-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BollweinOnTheHard18_9-0\" class=\"reference\"><a href=\"#cite_note-BollweinOnTheHard18-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CirianiFrag09_10-0\" class=\"reference\"><a href=\"#cite_note-CirianiFrag09-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CirianiKeep09_11-0\" class=\"reference\"><a href=\"#cite_note-CirianiKeep09-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-diVimercatiFrag14_12-0\" class=\"reference\"><a href=\"#cite_note-diVimercatiFrag14-12\">[12]<\/a><\/sup>\n<\/p><p>In this paper, we review the current and emerging research on privacy and confidentiality concerns in data outsourcing and highlight research directions in this field. In summary, our systematic review addresses security concerns in cloud database systems for both communicating and non-communicating servers. We also survey this research field in relation to the inference problem and the unresolved problems that are introduced. Recognizing these challenges, this paper provides an overview of our proposed (because this is an ongoing work) solution. The crux of that solution is to firstly optimize data distribution without the need to query the workload, then partition the database in the cloud by taking into consideration access control policies and data utility, before finally running a query evaluation model on a big data framework to securely process distributed queries while retaining access control. \n<\/p><p>The reminder of this paper is organized as follows. The next section describes the literature review methodology adopted in this paper. After that, we review emerging research on data outsourcing in the context of privacy concerns and data utility. Then we discuss data outsourcing in relation to the inference problem. Afterwards, we introduce our proposed solution to implement a secure distributed cloud database on a big data framework (Apache Spark). We close with future research directions and challenges, as well as our final conclusions.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Literature_review_methodology\">Literature review methodology<\/span><\/h2>\n<p>The methodology for literature review adopted in this paper follows the checklist proposed by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.<sup id=\"rdp-ebb-cite_ref-MoherPref09_13-0\" class=\"reference\"><a href=\"#cite_note-MoherPref09-13\">[13]<\/a><\/sup> It includes, as shown in Figure 1, three steps: input literature, processing steps, and review output.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Jebali_JofInfoTelec2020_5-1.jpg\" class=\"image wiki-link\" data-key=\"7f3b2a711a85a1d475e7bfffae354fa6\"><img alt=\"Fig1 Jebali JofInfoTelec2020 5-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/7a\/Fig1_Jebali_JofInfoTelec2020_5-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 1.<\/b> Three stages of the literature review process.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Input_literature\">Input literature<\/span><\/h3>\n<p>In this section we describe selected literature and their selection process. Firstly, our advance keyword research was conducted on the Google Scholar search engine, with a time filter from January 1, 1990 to December 31, 2019. Table 1 lists keywords used in different Google Scholar queries.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> Keywords used in our literature review search.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Keyword\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Number of viewed papers\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Access control, Data outsourcing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">43\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cloud computing, Authorization policies\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">78\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Database, inference leakage\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Confidentiality constraints, Cloud database\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">41\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Secure data integration\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Big data, Distributed query processing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">39\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Privacy, data publishing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The logical operator used between keywords during search was the \"And\" operator. Finally, from the 269 viewed papers, 43 articles were retained for review. Figure 2 shows their distribution by publication year.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Jebali_JofInfoTelec2020_5-1.jpg\" class=\"image wiki-link\" data-key=\"d37719a4a30b6e6f9d5ff2052f926aba\"><img alt=\"Fig2 Jebali JofInfoTelec2020 5-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/12\/Fig2_Jebali_JofInfoTelec2020_5-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 2.<\/b> Distribution of the 43 articles retained for review, by publication year.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Processing_steps\">Processing steps<\/span><\/h3>\n<p>During the review, papers were processed by identifying the problem, understanding the proposed solution process, and listing the important findings. We summarized and compared each paper with the papers associated with the similar problem. Then for each processed paper, three or four critical sentences were introduced to highlight the limits and specify potential directions that may be followed to enhance the proposed approaches. Based on our literature review, we classified the data outsourcing and access control papers into three categories, as shown in Figure 3. The first category of papers addresses the problem of secure data outsourcing when the servers in the cloud are unaware of each other. The second category addresses secure data outsourcing where interaction between servers exists and how this later can aggravate the situation. In the last category, we address data outsourcing in relation to the inference problem, as this later can exploit semantic constraints to bypass authorization policies at the cloud level.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Jebali_JofInfoTelec2020_5-1.jpg\" class=\"image wiki-link\" data-key=\"c203a1cd0fbba727f748a7ab35560e86\"><img alt=\"Fig3 Jebali JofInfoTelec2020 5-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/27\/Fig3_Jebali_JofInfoTelec2020_5-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 3.<\/b> Classification scheme of the literature retained for review.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Review_output\">Review output<\/span><\/h3>\n<p>The outcome of the methodological review process is presented later in this paper as our proposed solution. In the \"Proposed solution\" section, we present an incremental approach composed of three steps, each step treating one of the three problems mentioned in the previous subsection. We believe that our proposed solution is capable of providing good results compared to other reviewed approaches. Afterwards, we report other potential future research areas and challenges.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Preserving_confidentiality_in_data_outsourcing_scenarios\">Preserving confidentiality in data outsourcing scenarios<\/span><\/h2>\n<p>There is a consensus in the security research community about the efficiency of data outsourcing for solving data management problems.<sup id=\"rdp-ebb-cite_ref-SamaratiData10_2-1\" class=\"reference\"><a href=\"#cite_note-SamaratiData10-2\">[2]<\/a><\/sup> This consists of moving data from in-house storage to cloud databases, while also maintaining a balance between data confidentiality and utility (Figure 4).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Jebali_JofInfoTelec2020_5-1.jpg\" class=\"image wiki-link\" data-key=\"3e8beb1ec7f2d2823afdda28d9773c77\"><img alt=\"Fig4 Jebali JofInfoTelec2020 5-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/23\/Fig4_Jebali_JofInfoTelec2020_5-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 4.<\/b> A representation of secure data outsourcing.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>CSPs are considered honest-but-curious: the database servers answer user queries correctly and do not manage stored data, but they attempt to intelligently analyze data and queries in order to learn as much information as possible from them.\n<\/p><p>Two powerful techniques have been proposed to enforce access control in cloud databases. The first technique exploits vertical database fragmentation to keep some sensitive data separated from each other. The second technique resorts to encryption to make a single attribute invisible to unauthorized users. These two techniques can be implemented using the following approaches:\n<\/p>\n<ul><li> <b>Full outsourcing<\/b>: The entire in-house database is moved to the cloud. It considers vertical database fragmentation to enforce confidentiality constraints with more than two attributes by keeping them separated from each other among distributed servers. Moreover, it resorts to encryption in order to hide confidentiality constraints with a single attribute.<sup id=\"rdp-ebb-cite_ref-AggarwalTwoCan05_6-1\" class=\"reference\"><a href=\"#cite_note-AggarwalTwoCan05-6\">[6]<\/a><\/sup><\/li><\/ul>\n<ul><li> \"<b>Keep a few<\/b>\": This approach departs from encryption by involving the owner side. The attributes to be encrypted are stored in plain text on the owner side since this later is considered a trusted part. The rest of the database is distributed among servers while maintaining data confidentiality through vertical fragmentation.<sup id=\"rdp-ebb-cite_ref-CirianiKeep09_11-1\" class=\"reference\"><a href=\"#cite_note-CirianiKeep09-11\">[11]<\/a><\/sup><\/li><\/ul>\n<p>Aside from the fact that encrypting data for storing them externally carries a considerable cost<sup id=\"rdp-ebb-cite_ref-SamaratiData10_2-2\" class=\"reference\"><a href=\"#cite_note-SamaratiData10-2\">[2]<\/a><\/sup>, previous studies have primarily concentrated on non-communicating cloud servers.<sup id=\"rdp-ebb-cite_ref-AggarwalTwoCan05_6-2\" class=\"reference\"><a href=\"#cite_note-AggarwalTwoCan05-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CirianiFrag09_10-1\" class=\"reference\"><a href=\"#cite_note-CirianiFrag09-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CirianiKeep09_11-2\" class=\"reference\"><a href=\"#cite_note-CirianiKeep09-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-diVimercatiFrag14_12-1\" class=\"reference\"><a href=\"#cite_note-diVimercatiFrag14-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-diVmercatiFrag10_14-0\" class=\"reference\"><a href=\"#cite_note-diVmercatiFrag10-14\">[14]<\/a><\/sup> In this situation, servers are unaware of each other and do not exchange any information. When a master node receives a query, it decomposes it and processes it locally without the need to perform a join query. In recent years, researchers have studied the effect of communication between servers on query execution, and secure query evaluation strategies have been elaborated.<sup id=\"rdp-ebb-cite_ref-BkakriaPreserv13_4-1\" class=\"reference\"><a href=\"#cite_note-BkakriaPreserv13-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BkakriaConfid13_15-0\" class=\"reference\"><a href=\"#cite_note-BkakriaConfid13-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-diVimercatiEffic16_16-0\" class=\"reference\"><a href=\"#cite_note-diVimercatiEffic16-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-diVimercatiInteg13_17-0\" class=\"reference\"><a href=\"#cite_note-diVimercatiInteg13-17\">[17]<\/a><\/sup> \n<\/p><p>In the rest of this section, we discuss current and emerging research efforts in the first two of the three mentioned architectures. The third, inference control in data outsourcing, will be discussed in the following section.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Secure_data_outsourcing_with_non-communicating_servers\">Secure data outsourcing with non-communicating servers<\/span><\/h3>\n<p>In 2005, Aggarwal <i>et al.<\/i> presented some of the earliest research attempting to enforce access control in database outsourcing using vertical fragmentation.<sup id=\"rdp-ebb-cite_ref-AggarwalTwoCan05_6-3\" class=\"reference\"><a href=\"#cite_note-AggarwalTwoCan05-6\">[6]<\/a><\/sup> Under the assumption that servers do not communicate, the work aimed to split the database on two untrusted servers while preserving data privacy, with some of the attributes possibly encrypted. They demonstrated a secure fragmentation of a relation R is a triple (<i>F<\/i><sub>1<\/sub> , <i>F<\/i><sub>2<\/sub> , <i>E<\/i>) where <i>F<\/i><sub>1<\/sub> , <i>F<\/i><sub>2<\/sub> contain attributes in plain text stored in different servers and <i>E<\/i> is the set of encrypted attributes. The tuple identifier and the encrypted attributes were replicated with each fragment. The protection measures were also augmented by a query evaluation technique defining how queries on the original table can be transformed into queries on the fragmented table. \n<\/p><p>The work of Hudic <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-HudicData13_18-0\" class=\"reference\"><a href=\"#cite_note-HudicData13-18\">[18]<\/a><\/sup> introduces an approach to enforce confidentiality and privacy while outsourcing data to a CSP. The proposed technique relies on vertical fragmentation and applies only minimal encryption to preserve data exposure to malicious parties. However, the fragmentation algorithm enforces the database logic schema to be in a third normal form to produce a good fragmentation design, and the query execution cost was not proven to be minimal. \n<\/p><p>In 2007, Ciriani <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-CirianiFrag07_5-1\" class=\"reference\"><a href=\"#cite_note-CirianiFrag07-5\">[5]<\/a><\/sup> addressed the problem of privacy preserving data outsourcing by resorting to the combination of fragmentation and encryption. The former is exploited to break sensitive associations between attributes, while the latter enforces the privacy of singleton confidentiality constraints. The authors go on to define a formal model of minimal fragmentation and propose a heuristic minimal fragmentation algorithm to efficiently execute queries over fragments while preserving security properties. However, when a query executed over a fragment involves an attribute that is encrypted, an additional query is executed to evaluate the conditions of the attribute, leading to performance degradation by slowing down query processing.\n<\/p><p>In 2011, Ciriani <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-CirianiEnforcing11_19-0\" class=\"reference\"><a href=\"#cite_note-CirianiEnforcing11-19\">[19]<\/a><\/sup> addressed the concept of secure data publishing in the presence of confidentiality and visibility constraints. By modeling these two constraints as Boolean formulas and fragment as complete truth assignments, the authors rely on the Ordered Binary Decision Diagrams (OBDD) technique to check whether a fragmentation satisfies confidentiality and visibility constraints. The proposed algorithm runs using OBDD and returns a fragmentation that guarantees correctness and minimality. However, query execution cost was not investigated in this paper, and the algorithm runs only on a database schema with a single relation. \n<\/p><p>Xu <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-XuDatabase15_1-1\" class=\"reference\"><a href=\"#cite_note-XuDatabase15-1\">[1]<\/a><\/sup> studied the problem of finding secure fragmentation with minimum cost for query support in 2015. Firstly, they define the cost of a fragmentation <i>F<\/i> as the sum of the cost of each query <i>Qi<\/i> executed on <i>F<\/i> multiplied by the execution frequency of <i>Qi<\/i>. Secondly, they resort to using a heuristic local search graph-based approach to obtain near optimal fragmentation. The search space was modeled as a fragmentation graph, and transformation between fragmentation as a set of edges <i>E<\/i>. Then, two search strategies where proposed: a static search strategy, which is invariant with the number of steps in a solution path, as well as a dynamic search strategy based on guided local search, which guarantees the safeness of the final solution while avoiding a dead-end. However, this paper does not investigate visibility constraints, which is an important concept for data utility. Moreover, other heuristic search techniques could have been addressed (e.g., Tabu search or simulated annealing).\n<\/p><p>The 2009 work of Ciriani <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-CirianiKeep09_11-3\" class=\"reference\"><a href=\"#cite_note-CirianiKeep09-11\">[11]<\/a><\/sup> puts forward a new paradigm to securely publishing data in the cloud while completely departing from encryption, since encryption is sometimes considered a very rigid tool that is delicate in its configuration, while potentially slowing down query processing. The idea behind this work is to engage the owner side (assumed to be a trusted party) to store a limited portion of data (that is supposed to be encrypted) in the clear and use vertical fragmentation to break sensitive associations among data to be stored in the cloud. Their proposed algorithm computes a fragmentation solution that minimizes the load for the data owner while guaranteeing privacy concerns. Moreover, authors highlight other metrics that can be used to characterize the quality of a fragmentation and decide which attribute is affected to the client side and which attribute is externalized. However, engaging the client to enforce access control requires mediating every query in the system, which could lead to bottlenecks and negatively impact performance. \n<\/p><p>In 2017, Bollwein and Wiese<sup id=\"rdp-ebb-cite_ref-BollweinSepar17_8-1\" class=\"reference\"><a href=\"#cite_note-BollweinSepar17-8\">[8]<\/a><\/sup> proposed a separation of duties technique based on vertical fragmentation to address the problem of preserving confidentiality when outsourcing data to a CSP. To ensure privacy requirements were met, confidentiality constraints and data dependencies were introduced. The separation of duties problem was treated as an optimization problem to maximize the utility of the fragmented database and to enhance query execution over the distributed servers. However, the optimization problem was addressed only from the point of minimizing the number of distributed servers. Additionally, when collaboration between servers is established, the separation of duties approach is no longer efficient to preserve confidentiality constraints. The NP-hardness proofness of the separation of duties problem discussed in Bollwein and Wiese<sup id=\"rdp-ebb-cite_ref-BollweinSepar17_8-2\" class=\"reference\"><a href=\"#cite_note-BollweinSepar17-8\">[8]<\/a><\/sup> was later proven by the authors the following year.<sup id=\"rdp-ebb-cite_ref-BollweinOnTheHard18_9-1\" class=\"reference\"><a href=\"#cite_note-BollweinOnTheHard18-9\">[9]<\/a><\/sup> The separation of duties problem was addressed as an optimization problem by the combination of the two famous NP-hardness problems: bin packing and vertex coloring. The bin packing problem was introduced to take into consideration the capacity constraints of the servers, with the view that fragments should be placed in a minimum number of servers without exceeding the maximum capacity. Meanwhile, vertex coloring was introduced to enforce confidentiality constraints, seeing that the association of certain attributes in the same server violates confidentiality propriety. We should note, however, that this paper studies the separation of duties problem for single-relation databases, and to make the theory applicable in practical scenarios, a many-relations database should be used.\n<\/p><p>Keeping in mind the fact that communication between distributed servers in data outsourcing scenarios exacerbates privacy concerns, secure query evaluation strategies should be adopted. In the next subsection we investigate prior research on secure data outsourcing with communicating servers.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Secure_data_outsourcing:_The_case_of_communicating_servers\">Secure data outsourcing: The case of communicating servers<\/span><\/h3>\n<p>Over the past few years, some researchers have also investigated the problem of data outsourcing with communicating servers.<sup id=\"rdp-ebb-cite_ref-BkakriaPreserv13_4-2\" class=\"reference\"><a href=\"#cite_note-BkakriaPreserv13-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BkakriaConfid13_15-1\" class=\"reference\"><a href=\"#cite_note-BkakriaConfid13-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-diVimercatiEffic16_16-1\" class=\"reference\"><a href=\"#cite_note-diVimercatiEffic16-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-diVimercatiInteg13_17-1\" class=\"reference\"><a href=\"#cite_note-diVimercatiInteg13-17\">[17]<\/a><\/sup> Aside from attempting to guarantee confidentiality and privacy preservation when moving databases to the cloud, these works also implemented secure query evaluation strategies to retain the overall access control policy when servers communicate with each other. It is clear that when servers (containing sensitive attributes whose association is forbidden) interact through join queries, a user\u2019s privacy will be at risk. As such, secure query evaluation strategies aim to prevent the linking of sensitive attributes attempted by malicious actors.\n<\/p><p>Building on previous work<sup id=\"rdp-ebb-cite_ref-BkakriaPreserv13_4-3\" class=\"reference\"><a href=\"#cite_note-BkakriaPreserv13-4\">[4]<\/a><\/sup>, Bkakria <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BkakriaConfid13_15-2\" class=\"reference\"><a href=\"#cite_note-BkakriaConfid13-15\">[15]<\/a><\/sup> propose an approach that securely outsources data based on fragmentation and encryption. It also enforces access control when querying data by resorting to the query privacy technique. The approach examined the case of a many-relations database with new inter-table confidentiality constraints. The approach assumed that distributed servers could collude to break data confidentiality, and as such the connection between servers was intended to be based on a primary-key or foreign key. Additionally, the query evaluation model, which is based on private information retrieval, ensures sensitive attributes remain unlinkable by malicious actors using a semi-join query. However, their proposed technique enforces database schema to be normalized, and it generates a huge number of confidentiality constraints due to the transformation of both inter-table constraints to singleton and association constraints, which could affect the quality of the fragmentation algorithm. More generic queries should be considered.\n<\/p><p>A join query integrity check was tackled in the 2016 work of di Vimercati <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-diVimercatiEffic16_16-2\" class=\"reference\"><a href=\"#cite_note-diVimercatiEffic16-16\">[16]<\/a><\/sup> Inspired by prior work<sup id=\"rdp-ebb-cite_ref-diVimercatiInteg13_17-2\" class=\"reference\"><a href=\"#cite_note-diVimercatiInteg13-17\">[17]<\/a><\/sup>, the authors proposed a new technique for verifying the integrity of join queries computed by potentially untrusted cloud providers. The authors also aimed to prevent servers from learning from answered queries, which could lead to a breach of users' privacy. To do so, the first showed how markers, twins, salts, and buckets can be adapted to preserve integrity when a join query is executed as a semi-join. They then introduce two strategies to minimize the size of the verification: limit the adoption of buckets and salts to twins and markers only, and represent twins and markers through slim tuples. Additionally, the authors demonstrate through their experiments how the computational and communication overhead can be limited with an integrity check.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h3>\n<p>To summarize, we can classify the previously discussed approaches according to the following criterion: confidentiality constraints support, optimal distribution support, and secure query evaluation strategy support. We would like to mention that optimal distribution is treated through secure distributions that guarantee minimum query execution costs over fragments. From this point, it is clear that all mentioned approaches support access control verification through confidentiality constraints. However, query evaluations have not been tackled in all works.<sup id=\"rdp-ebb-cite_ref-BollweinSepar17_8-3\" class=\"reference\"><a href=\"#cite_note-BollweinSepar17-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CirianiFrag07_5-2\" class=\"reference\"><a href=\"#cite_note-CirianiFrag07-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CirianiKeep09_11-4\" class=\"reference\"><a href=\"#cite_note-CirianiKeep09-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-diVmercatiFrag10_14-1\" class=\"reference\"><a href=\"#cite_note-diVmercatiFrag10-14\">[14]<\/a><\/sup> Those approaches differ from the fact that some of them ensure minimum query execution costs and data utility for the database application, but other ones addressed the problem of data outsourcing with confidentiality constraints only. However, among the secure database distribution with query evaluation strategies, we find the work of Bkakria <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BkakriaPreserv13_4-4\" class=\"reference\"><a href=\"#cite_note-BkakriaPreserv13-4\">[4]<\/a><\/sup> provides an integral framework ensuring secure database fragmentation and communication between distributed servers. Also, it shows a reasonable query execution cost.\n<\/p><p>Nevertheless, Bkakria <i>et al.<\/i> assume that the threat comes from the cloud service providers that try to collude to break sensitive association between attributes. It does not address the case of an internal threat, where a malicious user aims to bypass access control with an inference channel. As such, we present an insightful discussion about data outsourcing in the presence of the inference problem in the following section.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Data_outsourcing_and_the_inference_problem\">Data outsourcing and the inference problem<\/span><\/h2>\n<p>Access control models protect sensitive data from direct disclosure via direct accesses; however, they fail to prevent indirect accesses.<sup id=\"rdp-ebb-cite_ref-FarkasTheInf02_20-0\" class=\"reference\"><a href=\"#cite_note-FarkasTheInf02-20\">[20]<\/a><\/sup> Indirect accesses via inference channels occurs when a malicious user combines the legitimate response that he received from the system with metadata (Figure 5). According to Guarnieri <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-GuarnieriSecur17_21-0\" class=\"reference\"><a href=\"#cite_note-GuarnieriSecur17-21\">[21]<\/a><\/sup>, types of external information that can be combined with legitimate data in order to produce an inference channel include database schema, system semantics, statistical information, exceptions, error messages, user-defined functions, and data dependencies.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Jebali_JofInfoTelec2020_5-1.jpg\" class=\"image wiki-link\" data-key=\"96acf652b9d2182afa987511ab8aee94\"><img alt=\"Fig5 Jebali JofInfoTelec2020 5-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/af\/Fig5_Jebali_JofInfoTelec2020_5-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 5.<\/b> An example of bypassing access control using an inference channel.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Although access control and inference control share the same goal of preventing unauthorized data disclosure, they differ in several fundamental aspects.<sup id=\"rdp-ebb-cite_ref-KatosAFrame11_22-0\" class=\"reference\"><a href=\"#cite_note-KatosAFrame11-22\">[22]<\/a><\/sup>\n<\/p><p>Table 2 highlights the major differences between them. According to our comparison in Table 2, we find that access control is more preferable than inference control from a complexity perspective. Consequently, several researchers have attempted to replace inference control engines with access control mechanisms. We refer the interested reader to Biskup <i>et al.<\/i> (2008)<sup id=\"rdp-ebb-cite_ref-BiskupReducing08_23-0\" class=\"reference\"><a href=\"#cite_note-BiskupReducing08-23\">[23]<\/a><\/sup>, Biskup <i>et al.<\/i> (2010)<sup id=\"rdp-ebb-cite_ref-BiskupEfficient10_24-0\" class=\"reference\"><a href=\"#cite_note-BiskupEfficient10-24\">[24]<\/a><\/sup>, and Katos <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-KatosAFrame11_22-1\" class=\"reference\"><a href=\"#cite_note-KatosAFrame11-22\">[22]<\/a><\/sup>, as the discussion of these approaches is beyond the scope of this paper.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 2.<\/b> Access control vs. inference control.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Access control\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Inference control\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Direct access control\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Indirect access control\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Deterministic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Related to stochastic channels\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Static, through a set of rules\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dynamic, varied through time and influenced by user action and queries\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Normal expense\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">More expensive then access control\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Computational efficiency and high accuracy of security control\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Efficiency and accuracy less than found with access control\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Modular and capable of covering a distributed data environment\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Adaptability to data distribution requires complicated techniques\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Inference_attacks_and_prevention_methods\">Inference attacks and prevention methods<\/span><\/h3>\n<p>According to Farkas and Jajodia<sup id=\"rdp-ebb-cite_ref-FarkasTheInf02_20-1\" class=\"reference\"><a href=\"#cite_note-FarkasTheInf02-20\">[20]<\/a><\/sup>, there are three types of inference attacks: a statistical attack, semantic attack, and inference attack due to <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_mining\" title=\"Data mining\" class=\"wiki-link\" data-key=\"be09d3680fe1608addedf6f62692ee47\">data mining<\/a>. For each of the mentioned techniques, researchers have devoted significant efforts to dealing with the inference problem. For statistical attacks, techniques like anonymization and data perturbation have been developed to protect data from indirect access. For security threats based on data mining, techniques like privacy-preserving data mining and privacy-preserving data publishing were carried out. Furthermore, additional research has also examined semantic attacks.<sup id=\"rdp-ebb-cite_ref-BrodskySecure00_25-0\" class=\"reference\"><a href=\"#cite_note-BrodskySecure00-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChenData06_26-0\" class=\"reference\"><a href=\"#cite_note-ChenData06-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SuControl91_27-0\" class=\"reference\"><a href=\"#cite_note-SuControl91-27\">[27]<\/a><\/sup> Among the literature can be found more than one criterion to classify approaches that deal with inference. One proposed criterion is to classify these approaches according to data and schema level.<sup id=\"rdp-ebb-cite_ref-YipData98_28-0\" class=\"reference\"><a href=\"#cite_note-YipData98-28\">[28]<\/a><\/sup> In such a classification, inference constraints are then classified into a schema constraints level or a data constraints level. Another criterion could classify according to the time when the inference control techniques are performed. According to this criterion, the proposed approaches are classified into two categories: design time<sup id=\"rdp-ebb-cite_ref-DelugachWizard96_29-0\" class=\"reference\"><a href=\"#cite_note-DelugachWizard96-29\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HinkeAerie92_30-0\" class=\"reference\"><a href=\"#cite_note-HinkeAerie92-30\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RathATool96_31-0\" class=\"reference\"><a href=\"#cite_note-RathATool96-31\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WangResist17_32-0\" class=\"reference\"><a href=\"#cite_note-WangResist17-32\">[32]<\/a><\/sup> and query run time<sup id=\"rdp-ebb-cite_ref-BrodskySecure00_25-1\" class=\"reference\"><a href=\"#cite_note-BrodskySecure00-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChenData06_26-1\" class=\"reference\"><a href=\"#cite_note-ChenData06-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AnDynam06_33-0\" class=\"reference\"><a href=\"#cite_note-AnDynam06-33\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ThuraisinghamDesign93_34-0\" class=\"reference\"><a href=\"#cite_note-ThuraisinghamDesign93-34\">[34]<\/a><\/sup>\n<\/p><p>The purpose of inference control at \"design time\" is to detect inference channels early on and eliminate them. This approach provides better performance for the system since no monitoring module is needed when the users query the database, as a consequence improving query execution time. Nevertheless, design time approaches are too restrictive and may lead to over-classification of the data. Additionally, it requires that the designer has a firm concept of how the system will be utilized. On the other hand, \"run time\" approaches provide data availability since they monitor the suspicious queries at run time. However, run time approaches lead to performance degradation of the database server since every query needs to be checked by the inference engine. Furthermore, the inference engine needs to manage a huge number of log files and users. As a result, this could slow down query processing. In addition, run time approaches could induce a non-deterministic access control behavior (e.g., users with the same privileges may not get the same response).\n<\/p><p>From this perspective, we can conclude that the main evaluation criterion of these techniques is a trade-off between availability and system performance. Some works have been elaborated to overcome these problems, especially for run time approaches. For example, Yang <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-YangNewPara07_35-0\" class=\"reference\"><a href=\"#cite_note-YangNewPara07-35\">[35]<\/a><\/sup> developed a new paradigm of inference control with trusted computing to push the inference control from server side to client side in order to mitigate the bottleneck on the database server. Furthermore, Staddon<sup id=\"rdp-ebb-cite_ref-StaddonDynamic03_36-0\" class=\"reference\"><a href=\"#cite_note-StaddonDynamic03-36\">[36]<\/a><\/sup> developed a run time inference control technique that retains fast query processing. The idea behind this work was to make query processing time depend on the length of the inference channel instead of user query history.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Inference_control_in_cloud_data_integration_systems\">Inference control in cloud data integration systems<\/span><\/h3>\n<p>Data outsourcing and the inference problem is an area of research that has been investigated for many years.<sup id=\"rdp-ebb-cite_ref-diVimercatiFrag14_12-2\" class=\"reference\"><a href=\"#cite_note-diVimercatiFrag14-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BiskupOnThe11_37-0\" class=\"reference\"><a href=\"#cite_note-BiskupOnThe11-37\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-daSilvaInfer04_38-0\" class=\"reference\"><a href=\"#cite_note-daSilvaInfer04-38\">[38]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HaddadAccess14_39-0\" class=\"reference\"><a href=\"#cite_note-HaddadAccess14-39\">[39]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SayahInference15_40-0\" class=\"reference\"><a href=\"#cite_note-SayahInference15-40\">[40]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SellamiInfer15_41-0\" class=\"reference\"><a href=\"#cite_note-SellamiInfer15-41\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TuranGraph18_42-0\" class=\"reference\"><a href=\"#cite_note-TuranGraph18-42\">[42]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TuranSecure17_43-0\" class=\"reference\"><a href=\"#cite_note-TuranSecure17-43\">[43]<\/a><\/sup> Inference leakage is recognized as a major barrier to cloud computing and other data outsourcing or database-as-a-service arrangements. The problem is that the designer of the system cannot anticipate the inference channels that arise on the cloud level and could lead to security breaches. \n<\/p><p>Researchers like da Silva <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-daSilvaInfer04_38-1\" class=\"reference\"><a href=\"#cite_note-daSilvaInfer04-38\">[38]<\/a><\/sup> have been able to pinpoint the inference that occurs in a homogeneous peer agent through distributed data mining, calling this process a \"peer-to-peer agent-based data mining system.\" They assert that performing distributed data mining (DDM) in such extremely open distributed systems exacerbates data privacy and security issues. As a matter of fact, inference occurs in DDM when one or more peer sites learn any confidential information (e.g., a model, patterns, or data themselves) about the dataset owned by other peers during a data mining session. The authors classified inference attacks in DDM in two categories:\n<\/p>\n<ul><li> <b>Inside attack scenario<\/b>: This scenario occurs when a peer tries to infer sensitive information from other peers in the same mining group. Depending on the number of attackers, the authors make a distinction between a single attack (when one peer behaves maliciously) and a coalition attack (when many sites collude to attack one site). Moreover, a probe attack was introduced by the authors, which is independent of the number of peers participating in the attack.<\/li><\/ul>\n<ul><li> <b>Outside attack scenario<\/b>: This scenario takes place when a set of malicious peers try to infer useful information from other peers in a different mining group. In this case, an eavesdropping channel attack is performed by malicious peers to steal information from other peers.<\/li><\/ul>\n<p>After identifying DDM inference attacks, the authors propose an algorithm known as KDEC to control potential inside and outside attacks to particular schema for homogeneous distributed clustering. The main idea behind KDEC is to reconstruct the data from the kernel density estimates, given that a malicious peer can use the reconstruction algorithm to infer non-local data. However, the algorithm proposed by the authors needs to be improved from an accuracy point to expose further possible weakness of the KDEC schema.\n<\/p><p>Inference control in cloud integration systems has also been investigated in the last decade through the work of Haddad <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-HaddadAccess14_39-1\" class=\"reference\"><a href=\"#cite_note-HaddadAccess14-39\">[39]<\/a><\/sup>, Sayah <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SayahInference15_40-1\" class=\"reference\"><a href=\"#cite_note-SayahInference15-40\">[40]<\/a><\/sup>, and Sellami <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SellamiInfer15_41-1\" class=\"reference\"><a href=\"#cite_note-SellamiInfer15-41\">[41]<\/a><\/sup> In such systems, a mediator is defined as a unique entry point to the distributed data sources. It provides the user a unique view of the distributed data. From a security point of view, access control is a major challenge in this situation since the global policy of the mediator in the cloud level must comply with the back-end data source policies, in addition to possibly enforcing additional security properties (Figure 6). The problem is that the system (or the designer of the system) cannot anticipate the inference channels that arise due to the dependencies that appear at the mediator level.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Jebali_JofInfoTelec2020_5-1.jpg\" class=\"image wiki-link\" data-key=\"5a08c145a3570296d6fa4ea36ee8d89e\"><img alt=\"Fig6 Jebali JofInfoTelec2020 5-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/69\/Fig6_Jebali_JofInfoTelec2020_5-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 6.<\/b> Representation of a secure cloud data integration system.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The first known work attempting to control inference in data integration systems is by Haddad <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-HaddadAccess14_39-2\" class=\"reference\"><a href=\"#cite_note-HaddadAccess14-39\">[39]<\/a><\/sup> The authors propose an incremental approach to prevent inference with functional dependencies. The proposed methodology includes three steps:\n<\/p>\n<ol><li> <i>Synthesize global policies<\/i>: Derives the authorization rule of each virtual relation individually by the way that it preserves the local authorization of the local relations involved in the virtual relation.<\/li>\n<li> <i>Detect violations<\/i>: By resorting to a graph-based approach, this step aims at identifying all the violations that could occur using functional dependencies. Such violations are called \"transaction violations,\" consisting of a series of innocuous queries that lead to violation of authorization rules.<\/li>\n<li> <i>Reconfigure authorization policies<\/i>: The author proposes two methods to forbid the completion of each transaction violation. The first one uses a historical access control by keeping track of previous queries to evaluate the current query (this method is considered to be a run-time approach). The second one proposes to reconfigure the global authorization policies at the mediator level in a way that no authorization violation will occur (this method is considered to be at design-time of the global security policy).<\/li><\/ol>\n<p>It should be noted that Haddad <i>et al.<\/i> have discussed only semantic constraints due to functional dependencies. Neither inclusion nor multivalued dependencies were investigated. Additionally, other mapping approaches need to be discussed, such as LAV and GLAV approaches.\n<\/p><p>Inspired by Haddad <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-HaddadAccess14_39-3\" class=\"reference\"><a href=\"#cite_note-HaddadAccess14-39\">[39]<\/a><\/sup>, Sellami <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SellamiInfer15_41-2\" class=\"reference\"><a href=\"#cite_note-SellamiInfer15-41\">[41]<\/a><\/sup> propose an approach aiming to control inference in cloud integration systems. The proposed methodology resorts to formal concept analysis as a formal framework to reason about authorization rules, and functional dependencies as a source of inference. The authors adopt an access control model with authorization views and propose an incremental approach with three steps:\n<\/p>\n<ol><li>Generate the global policy, global schema, and global functional dependency (FD): This step takes as input a set of source schema together with their access control policies and starts by translating the schema and policies to formal contexts. Then, the global policy is generated in a way that the source rules are preserved at the global level. Next the schema of the mediator (virtual relations) is generated from the global policy to avoid useless attributes combinations (every attribute in the mediator schema is controlled by the global policy). Finally, a global FD is considered from the source FD as a formal context.<\/li>\n<li>Identify disclosure transactions: By resorting to formal concept analysis (FCA) as a framework to reason about the global policy, the authors identify the profiles to be denied from accessing sensitive attributes at the mediator level. Then, they extract the transaction violations by reasoning about the global FD.<\/li>\n<li>Reconfigure: This step is achieved by two methods. At design time, use a policy healing to complete the global policy, with additional rules to ensure that no transaction violation is achieved. At query run time, use a monitoring engine to prohibit suspicious queries.<\/li><\/ol>\n<p>Sayah <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SayahInference15_40-2\" class=\"reference\"><a href=\"#cite_note-SayahInference15-40\">[40]<\/a><\/sup> have examined inference that arises in the web through a Resource Description Framework (RDF) store. They propose a fine-grained framework for RDF data, and then they exploit a closed graph to verify the consistency property of an access control policy when inference rules and authorization rules interact. Without accessing the data (at policy design-time), the authors propose an algorithm to verify if an information leakage will arise given a policy <i>P<\/i> and a set of inference rules R. Furthermore, the authors demonstrate the applicability of the access control model using a conflict resolution strategy (most specific takes precedence).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Inference_control_in_distributed_cloud_database_systems\">Inference control in distributed cloud database systems<\/span><\/h3>\n<p>Biskup <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BiskupOnThe11_37-1\" class=\"reference\"><a href=\"#cite_note-BiskupOnThe11-37\">[37]<\/a><\/sup> resort to using a Controlled Query Evaluation strategy (CQE) to detect inference based on the knowledge of non-confidential information contained in the outsourced fragments and prior knowledge that a malicious user might have. As CQE relies on a logic-oriented view of database systems, the main idea of this approach is to model fragmentation in a logic-oriented fashion to allow for inference proofness to be proved formally despite the semantic database constraints that an attacker may hold. This type of vertical database fragmentation technique was also considered by di Vimercati <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-diVimercatiFrag14_12-3\" class=\"reference\"><a href=\"#cite_note-diVimercatiFrag14-12\">[12]<\/a><\/sup> to ensure data confidentiality in the presence of data dependencies among attributes. Those dependencies allow unauthorized users to deduce information about sensitive attributes. In this work, the authors define three types of confidentiality violation that can be caused by data dependencies. First, this can happen when a sensitive attribute or association is exposed by the attributes in a fragment. Second, this can also occur if an attribute appearing in a fragment is also derivable from some attributes in another fragment, thus enabling linkability among such fragments. Finally, this can occur when an attribute is derivable (independently) from attributes appearing in different fragments, thus enabling linkability among these fragments. To tackle these issues, authors reformulate the problem graphically through a hyper-graph representation and then compute the closure of a fragmentation by deducing all information derivable from its fragments via dependencies to identify indirect access. Nevertheless, the major limit of this approach is that it explores the problem only in a single-relation database.\n<\/p><p>Although data outsourcing was not explicitly mentioned by Turan<i> et al.<\/i><sup id=\"rdp-ebb-cite_ref-TuranGraph18_42-1\" class=\"reference\"><a href=\"#cite_note-TuranGraph18-42\">[42]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TuranSecure17_43-1\" class=\"reference\"><a href=\"#cite_note-TuranSecure17-43\">[43]<\/a><\/sup>, their two works from 2017 and 2018 aim to control the inference problem caused by functional dependencies and meaningful join proactively by decomposing the relational logical schema into a set of views (to be queried by the user) where inference channels cannot appear. In 2017, the authors proposed<sup id=\"rdp-ebb-cite_ref-TuranSecure17_43-2\" class=\"reference\"><a href=\"#cite_note-TuranSecure17-43\">[43]<\/a><\/sup> a proactive and decomposition-based inference control strategy for relational databases to prevent access to a forbidden set via direct or indirect access. The proposed decomposition algorithm controls both functional and probabilistic dependencies by breaking down those leading to the inference of a forbidden attribute set. However, this approach was considered too rigid for the fact that if the ways of associate forbidden set attributes are defined as a chain of functional dependencies, the algorithm breaks these chains from both sides for both attributes. Parting from this limit, the same researchers proposed a graph-based approach<sup id=\"rdp-ebb-cite_ref-TuranGraph18_42-2\" class=\"reference\"><a href=\"#cite_note-TuranGraph18-42\">[42]<\/a><\/sup> consisting of proactive decomposition of the external schema in order to satisfy both the forbidden and required associations of attributes. In this work, functional dependencies are represented as a graph in which vertices are attributes and edges are functional dependencies. The inference channel is then defined as a process of searching a sub-tree in the graph containing the attributes that need to be related. Compared to their prior approach<sup id=\"rdp-ebb-cite_ref-TuranSecure17_43-3\" class=\"reference\"><a href=\"#cite_note-TuranSecure17-43\">[43]<\/a><\/sup>, this method sees the cut of the inference channel getting relaxed by cutting the chains only at a single point, consequently minimizing dependency loss. Nevertheless, like the previous technique, it leads to semantic loss and a need to query rewriting techniques to query decomposed views.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Proposed_solution\">Proposed solution<\/span><\/h2>\n<p>In this section, we present an approach that relies on the relational model and aims to produce a set of secure sub-schemas, with each represented by a partition <i>P<sub>i<\/sub><\/i> stored exactly on one server by the CSP (Figure 7). This approach also introduces a secure distributed query evaluation strategy to efficiently request data from distributed partitions while retaining access control policies. To do that, our proposed methodology takes as input a set of functional dependencies (FD), as well as a relational schema R, and applies the following phases:\n<\/p>\n<ol><li> <i>Constraints generation<\/i>: This phase aims at generating two types of constraints that, in addition to the confidentiality constraints, will guide the process of vertical schema partitioning. This is done through two steps: visibility constraints generation and constraint-based inference control generation. During visibility constraints generation, the constraints are enforced as soft constraints in the partitioning phase, given that their severity is less than confidentiality constraints. To generate them, we perform a semantic analysis of the relational schema in order to detect semantic relatedness between attributes and users roles. These constraints should be preserved (stay visible) when the relational schema is fragmented (i.e., we aim to maximize intra-dependency between attributes that seem to be frequently accessed by the same role while minimizing the inter-dependency between attributes in separate parts). Then during constraint-based inference control generation, the constraints are enforced (like confidentiality constraints) as hard constraints. In this step, we resort to the method proposed by Turan <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TuranGraph18_42-3\" class=\"reference\"><a href=\"#cite_note-TuranGraph18-42\">[42]<\/a><\/sup> to build a functional dependencies graph and generate join chains. Then, we use a relaxed technique to cut the join chains only at a single point in order to minimize dependency loss. By cutting a join chain at a single point, the enforcement of the attributes in the left-hand side (LHS) and right-hand side (RHS) of the functional dependency can occur, with the cut point representing a confidentiality constraint. As a consequence, we guarantee that the join chain is broken.<\/li>\n<li> <i>Schema partitioning<\/i>: In this phase, we resort to hypergraph theory to represent the partitioning problem as a hypergraph constraint satisfaction problem. Then, we reformulate the problem as a multi-objective function F to be optimized. Therefore, we propose a greedy algorithm to partition the constrained hypergraph into <i>k<\/i> parts while minimizing the multi-objective function F.<\/li>\n<li> <i>Query evaluation model<\/i>: In this step, we propose a monitor module to mediate every query issued from users against data stored in distributed partitions. The monitor module is built on top of an Apache Spark system, which contains two mechanisms: a role-based access control mechanism and a history-based access control mechanism. The first mechanism checks for the user role of who issued the query and, if it is not granted to execute distributed query, then this later will be forwarded to the CSP. Otherwise, the query is forwarded to the history access control mechanism, which takes as input a set of transaction violations to be prohibited and checks if the cumulative of user past queries and the current query could complete a transaction violation. If that proves to be the case, the query is revoked.<\/li><\/ol>\n<p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Jebali_JofInfoTelec2020_5-1.jpg\" class=\"image wiki-link\" data-key=\"a95aa6aa914ab46c1b3691c0592aa05e\"><img alt=\"Fig7 Jebali JofInfoTelec2020 5-1.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d9\/Fig7_Jebali_JofInfoTelec2020_5-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 7.<\/b> A proposed methodology to generate secure partitions and lock suspicious queries.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Future_research_directions_and_challenges\">Future research directions and challenges<\/span><\/h2>\n<p>Since the discussed works are recent, there are a number of concepts associated to access control, privacy, data outsourcing, and database semantics which could be considered to ensure better data security and utility in the cloud. As such, there are several research directions to pursue going forward.\n<\/p><p>Functional dependencies should be considered a threat source in data outsourcing scenarios. Unlike the approaches by Turan <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TuranGraph18_42-4\" class=\"reference\"><a href=\"#cite_note-TuranGraph18-42\">[42]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TuranSecure17_43-4\" class=\"reference\"><a href=\"#cite_note-TuranSecure17-43\">[43]<\/a><\/sup>, we aim in our future work to prevent inference from occurring in a distributed cloud database. Our graph-based approach detects inference channels caused by functional dependencies and breaks those channels by exploiting vertical database fragmentation while minimizing dependency loss.\n<\/p><p>Prior researchers have only dealt with semantic constraints represented by functional and probabilistic dependencies as a source of inference. However, other semantic constraints\u2014e.g., inclusion dependencies, join dependencies, and multivalued dependencies\u2014should be considered as sources of inference.\n<\/p><p>Another future direction to examine is when the workload will become available after the database is set up in the cloud. The challenge here is in how to dynamically reallocate the distributed database fragments among distributed servers while retaining access control policy.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>In this work, we conducted a literature review of current and emerging research on privacy and confidentiality concerns in data outsourcing. We have introduced different research efforts to ensure user privacy in the database-as-a-service paradigm with both communicating and non-communicating servers. Additionally, an insightful discussion about inference control was introduced, as well as a proposed method of tackling the issue. We also pinpoint potential issues that are still unresolved. These issues are expected to be addressed in future work.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>No potential conflict of interest was reported by the author(s).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-XuDatabase15-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-XuDatabase15_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-XuDatabase15_1-1\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xu, X.; Xiong, L.; Liu, J. (2015). \"Database Fragmentation with Confidentiality Constraints: A Graph Search Approach\". <i>Proceedings of the 5th ACM Conference on Data and Application Security and Privacy<\/i>: 263\u201370. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2699026.2699121\" target=\"_blank\">10.1145\/2699026.2699121<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Database+Fragmentation+with+Confidentiality+Constraints%3A+A+Graph+Search+Approach&rft.jtitle=Proceedings+of+the+5th+ACM+Conference+on+Data+and+Application+Security+and+Privacy&rft.aulast=Xu%2C+X.%3B+Xiong%2C+L.%3B+Liu%2C+J.&rft.au=Xu%2C+X.%3B+Xiong%2C+L.%3B+Liu%2C+J.&rft.date=2015&rft.pages=263%E2%80%9370&rft_id=info:doi\/10.1145%2F2699026.2699121&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SamaratiData10-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SamaratiData10_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-SamaratiData10_2-1\">2.1<\/a><\/sup> <sup><a href=\"#cite_ref-SamaratiData10_2-2\">2.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Samarati, P.; di Vimarcati, S.D.C. (2010). \"Data protection in outsourcing scenarios: Issues and directions\". <i>Proceedings of the 5th ACM Symposium on Information, Computer and Communications Security<\/i>: 1\u201314. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1755688.1755690\" target=\"_blank\">10.1145\/1755688.1755690<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+protection+in+outsourcing+scenarios%3A+Issues+and+directions&rft.jtitle=Proceedings+of+the+5th+ACM+Symposium+on+Information%2C+Computer+and+Communications+Security&rft.aulast=Samarati%2C+P.%3B+di+Vimarcati%2C+S.D.C.&rft.au=Samarati%2C+P.%3B+di+Vimarcati%2C+S.D.C.&rft.date=2010&rft.pages=1%E2%80%9314&rft_id=info:doi\/10.1145%2F1755688.1755690&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BiskupData13-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BiskupData13_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Biskup, J.; Preu\u00df, M. (2013). \"Database Fragmentation with Encryption: Under Which Semantic Constraints and A Priori Knowledge Can Two Keep a Secret?\". <i>Data and Applications Security and Privacy XXVIII<\/i>: 17\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-39256-6_2\" target=\"_blank\">10.1007\/978-3-642-39256-6_2<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Database+Fragmentation+with+Encryption%3A+Under+Which+Semantic+Constraints+and+A+Priori+Knowledge+Can+Two+Keep+a+Secret%3F&rft.jtitle=Data+and+Applications+Security+and+Privacy+XXVIII&rft.aulast=Biskup%2C+J.%3B+Preu%C3%9F%2C+M.&rft.au=Biskup%2C+J.%3B+Preu%C3%9F%2C+M.&rft.date=2013&rft.pages=17%E2%80%9332&rft_id=info:doi\/10.1007%2F978-3-642-39256-6_2&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BkakriaPreserv13-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BkakriaPreserv13_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-BkakriaPreserv13_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-BkakriaPreserv13_4-2\">4.2<\/a><\/sup> <sup><a href=\"#cite_ref-BkakriaPreserv13_4-3\">4.3<\/a><\/sup> <sup><a href=\"#cite_ref-BkakriaPreserv13_4-4\">4.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bkakria, A.; Cuppens, F.; Cuppens-Boulahia, N. et al. (2013). \"Preserving Multi-relational Outsourced Databases Confidentiality using Fragmentation and Encryption\". <i>JoWUA<\/i> <b>4<\/b> (2): 39\u201362. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.22667%2FJOWUA.2013.06.31.039\" target=\"_blank\">10.22667\/JOWUA.2013.06.31.039<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Preserving+Multi-relational+Outsourced+Databases+Confidentiality+using++Fragmentation+and+Encryption&rft.jtitle=JoWUA&rft.aulast=Bkakria%2C+A.%3B+Cuppens%2C+F.%3B+Cuppens-Boulahia%2C+N.+et+al.&rft.au=Bkakria%2C+A.%3B+Cuppens%2C+F.%3B+Cuppens-Boulahia%2C+N.+et+al.&rft.date=2013&rft.volume=4&rft.issue=2&rft.pages=39%E2%80%9362&rft_id=info:doi\/10.22667%2FJOWUA.2013.06.31.039&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CirianiFrag07-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CirianiFrag07_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-CirianiFrag07_5-1\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-CirianiFrag07_5-2\">5.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ciriani, V.; di Vimaercati, S.D.C.; Foresti, S. et al. (2007). \"Fragmentation and Encryption to Enforce Privacy in Data Storage\". <i>Computer Security - ESORICS 2007<\/i>: 171\u201386. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-540-74835-9_12\" target=\"_blank\">10.1007\/978-3-540-74835-9_12<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fragmentation+and+Encryption+to+Enforce+Privacy+in+Data+Storage&rft.jtitle=Computer+Security+-+ESORICS+2007&rft.aulast=Ciriani%2C+V.%3B+di+Vimaercati%2C+S.D.C.%3B+Foresti%2C+S.+et+al.&rft.au=Ciriani%2C+V.%3B+di+Vimaercati%2C+S.D.C.%3B+Foresti%2C+S.+et+al.&rft.date=2007&rft.pages=171%E2%80%9386&rft_id=info:doi\/10.1007%2F978-3-540-74835-9_12&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AggarwalTwoCan05-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AggarwalTwoCan05_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-AggarwalTwoCan05_6-1\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-AggarwalTwoCan05_6-2\">6.2<\/a><\/sup> <sup><a href=\"#cite_ref-AggarwalTwoCan05_6-3\">6.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aggarwal, G.; Bawa, M.; Ganesan, P. et al. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ilpubs.stanford.edu:8090\/659\/\" target=\"_blank\">\"Two Can Keep a Secret: A Distributed Architecture for Secure Database Services\"<\/a>. <i>Second Biennial Conference on Innovative Data Systems Research<\/i>: 1\u201314<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ilpubs.stanford.edu:8090\/659\/\" target=\"_blank\">http:\/\/ilpubs.stanford.edu:8090\/659\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Two+Can+Keep+a+Secret%3A+A+Distributed+Architecture+for+Secure+Database+Services&rft.jtitle=Second+Biennial+Conference+on+Innovative+Data+Systems+Research&rft.aulast=Aggarwal%2C+G.%3B+Bawa%2C+M.%3B+Ganesan%2C+P.+et+al.&rft.au=Aggarwal%2C+G.%3B+Bawa%2C+M.%3B+Ganesan%2C+P.+et+al.&rft.date=2005&rft.pages=1%E2%80%9314&rft_id=http%3A%2F%2Filpubs.stanford.edu%3A8090%2F659%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlsirhaniImprov17-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlsirhaniImprov17_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alsirhani, A.; Bodorik, P. Sampalli, S. (2017). \"Improving Database Security in Cloud Computing by Fragmentation of Data\". <i>Proceedings of the 2017 International Conference on Computer and Applications<\/i>: 43\u201349. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCOMAPP.2017.8079737\" target=\"_blank\">10.1109\/COMAPP.2017.8079737<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improving+Database+Security+in+Cloud+Computing+by+Fragmentation+of+Data&rft.jtitle=Proceedings+of+the+2017+International+Conference+on+Computer+and+Applications&rft.aulast=Alsirhani%2C+A.%3B+Bodorik%2C+P.+Sampalli%2C+S.&rft.au=Alsirhani%2C+A.%3B+Bodorik%2C+P.+Sampalli%2C+S.&rft.date=2017&rft.pages=43%E2%80%9349&rft_id=info:doi\/10.1109%2FCOMAPP.2017.8079737&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BollweinSepar17-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BollweinSepar17_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-BollweinSepar17_8-1\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-BollweinSepar17_8-2\">8.2<\/a><\/sup> <sup><a href=\"#cite_ref-BollweinSepar17_8-3\">8.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bollwein, F.; Wiese, L. (2017). \"Separation of Duties for Multiple Relations in Cloud Databases as an Optimization Problem\". <i>Proceedings of the 21st International Database Engineering & Applications Symposium<\/i>: 98\u2013107. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3105831.3105873\" target=\"_blank\">10.1145\/3105831.3105873<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Separation+of+Duties+for+Multiple+Relations+in+Cloud+Databases+as+an+Optimization+Problem&rft.jtitle=Proceedings+of+the+21st+International+Database+Engineering+%26+Applications+Symposium&rft.aulast=Bollwein%2C+F.%3B+Wiese%2C+L.&rft.au=Bollwein%2C+F.%3B+Wiese%2C+L.&rft.date=2017&rft.pages=98%E2%80%93107&rft_id=info:doi\/10.1145%2F3105831.3105873&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BollweinOnTheHard18-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BollweinOnTheHard18_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-BollweinOnTheHard18_9-1\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bollwein, F.; Wiese, L. (2018). \"On the Hardness of Separation of Duties Problems for Cloud Databases\". <i>Proceedings of TrustBus 2018: Trust, Privacy and Security in Digital Business<\/i>: 23\u201338. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-98385-1_3\" target=\"_blank\">10.1007\/978-3-319-98385-1_3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=On+the+Hardness+of+Separation+of+Duties+Problems+for+Cloud+Databases&rft.jtitle=Proceedings+of+TrustBus+2018%3A+Trust%2C+Privacy+and+Security+in+Digital+Business&rft.aulast=Bollwein%2C+F.%3B+Wiese%2C+L.&rft.au=Bollwein%2C+F.%3B+Wiese%2C+L.&rft.date=2018&rft.pages=23%E2%80%9338&rft_id=info:doi\/10.1007%2F978-3-319-98385-1_3&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CirianiFrag09-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CirianiFrag09_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-CirianiFrag09_10-1\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ciriani, V.; di Vimercati, S.D.C.; Foresti, S. et al. (2009). \"Fragmentation Design for Efficient Query Execution over Sensitive Distributed Databases\". <i>Proceedings of the 29th IEEE International Conference on Distributed Computing Systems<\/i>: 32\u201339. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FICDCS.2009.52\" target=\"_blank\">10.1109\/ICDCS.2009.52<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fragmentation+Design+for+Efficient+Query+Execution+over+Sensitive+Distributed+Databases&rft.jtitle=Proceedings+of+the+29th+IEEE+International+Conference+on+Distributed+Computing+Systems&rft.aulast=Ciriani%2C+V.%3B+di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.+et+al.&rft.au=Ciriani%2C+V.%3B+di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.+et+al.&rft.date=2009&rft.pages=32%E2%80%9339&rft_id=info:doi\/10.1109%2FICDCS.2009.52&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CirianiKeep09-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CirianiKeep09_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-CirianiKeep09_11-1\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-CirianiKeep09_11-2\">11.2<\/a><\/sup> <sup><a href=\"#cite_ref-CirianiKeep09_11-3\">11.3<\/a><\/sup> <sup><a href=\"#cite_ref-CirianiKeep09_11-4\">11.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ciriani, V.; di Vimercati, S.D.C.; Foresti, S. et al. (2009). \"Keep a Few: Outsourcing Data While Maintaining Confidentiality\". <i>Computing Security - ESORICS 2009<\/i>: 440\u201355. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-04444-1_27\" target=\"_blank\">10.1007\/978-3-642-04444-1_27<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Keep+a+Few%3A+Outsourcing+Data+While+Maintaining+Confidentiality&rft.jtitle=Computing+Security+-+ESORICS+2009&rft.aulast=Ciriani%2C+V.%3B+di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.+et+al.&rft.au=Ciriani%2C+V.%3B+di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.+et+al.&rft.date=2009&rft.pages=440%E2%80%9355&rft_id=info:doi\/10.1007%2F978-3-642-04444-1_27&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-diVimercatiFrag14-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-diVimercatiFrag14_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-diVimercatiFrag14_12-1\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-diVimercatiFrag14_12-2\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-diVimercatiFrag14_12-3\">12.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">di Vimercati, S.D.C.; Foresti, S.; Jajodia, S. et al. (2014). \"Fragmentation in Presence of Data Dependencies\". <i>IEEE Transactions on Dependable and Secure Computing<\/i> <b>11<\/b> (6): 510\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTDSC.2013.2295798\" target=\"_blank\">10.1109\/TDSC.2013.2295798<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fragmentation+in+Presence+of+Data+Dependencies&rft.jtitle=IEEE+Transactions+on+Dependable+and+Secure+Computing&rft.aulast=di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.%3B+Jajodia%2C+S.+et+al.&rft.au=di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.%3B+Jajodia%2C+S.+et+al.&rft.date=2014&rft.volume=11&rft.issue=6&rft.pages=510%E2%80%9323&rft_id=info:doi\/10.1109%2FTDSC.2013.2295798&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoherPref09-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoherPref09_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moher, D.; Liberti, A.; Tetzlaff, J. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2707599\" target=\"_blank\">\"Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement\"<\/a>. <i>PLoS One<\/i> <b>6<\/b> (7): e1000097. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1000097\" target=\"_blank\">10.1371\/journal.pmed.1000097<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2707599\/\" target=\"_blank\">PMC2707599<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19621072\" target=\"_blank\">19621072<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2707599\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2707599<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Preferred+reporting+items+for+systematic+reviews+and+meta-analyses%3A+the+PRISMA+statement&rft.jtitle=PLoS+One&rft.aulast=Moher%2C+D.%3B+Liberti%2C+A.%3B+Tetzlaff%2C+J.+et+al.&rft.au=Moher%2C+D.%3B+Liberti%2C+A.%3B+Tetzlaff%2C+J.+et+al.&rft.date=2009&rft.volume=6&rft.issue=7&rft.pages=e1000097&rft_id=info:doi\/10.1371%2Fjournal.pmed.1000097&rft_id=info:pmc\/PMC2707599&rft_id=info:pmid\/19621072&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2707599&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-diVmercatiFrag10-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-diVmercatiFrag10_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-diVmercatiFrag10_14-1\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">di Vimercati, S.D.C.; Foresti, S.; Jajodia, S. et al. (2010). \"Fragments and loose associations: respecting privacy in data publishing\". <i>Proceedings of the VLDB Endowment<\/i> <b>3<\/b> (1\u20132). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.14778%2F1920841.1921009\" target=\"_blank\">10.14778\/1920841.1921009<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fragments+and+loose+associations%3A+respecting+privacy+in+data+publishing&rft.jtitle=Proceedings+of+the+VLDB+Endowment&rft.aulast=di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.%3B+Jajodia%2C+S.+et+al.&rft.au=di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.%3B+Jajodia%2C+S.+et+al.&rft.date=2010&rft.volume=3&rft.issue=1%E2%80%932&rft_id=info:doi\/10.14778%2F1920841.1921009&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BkakriaConfid13-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BkakriaConfid13_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-BkakriaConfid13_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-BkakriaConfid13_15-2\">15.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bkakria, A.; Cuppens, F.; Cuppens-Boulahia, N. et al. (2013). \"Confidentiality-Preserving Query Execution of Fragmented Outsourced Data\". <i>Proceeding of ICT-EurAsia 2013: Information and Communication Technology<\/i> <b>4<\/b> (2): 426-440. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-36818-9_47\" target=\"_blank\">10.1007\/978-3-642-36818-9_47<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Confidentiality-Preserving+Query+Execution+of+Fragmented+Outsourced+Data&rft.jtitle=Proceeding+of+ICT-EurAsia+2013%3A+Information+and+Communication+Technology&rft.aulast=Bkakria%2C+A.%3B+Cuppens%2C+F.%3B+Cuppens-Boulahia%2C+N.+et+al.&rft.au=Bkakria%2C+A.%3B+Cuppens%2C+F.%3B+Cuppens-Boulahia%2C+N.+et+al.&rft.date=2013&rft.volume=4&rft.issue=2&rft.pages=426-440&rft_id=info:doi\/10.1007%2F978-3-642-36818-9_47&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-diVimercatiEffic16-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-diVimercatiEffic16_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-diVimercatiEffic16_16-1\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-diVimercatiEffic16_16-2\">16.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">di Vimarcati, S.D.C.; Foresti, S.; Jajodia, S. et al. (2016). \"Efficient integrity checks for join queries in the cloud\". <i>Journal of Computer Security<\/i> <b>24<\/b> (3): 347\u201378. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2FJCS-160545\" target=\"_blank\">10.3233\/JCS-160545<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Efficient+integrity+checks+for+join+queries+in+the+cloud&rft.jtitle=Journal+of+Computer+Security&rft.aulast=di+Vimarcati%2C+S.D.C.%3B+Foresti%2C+S.%3B+Jajodia%2C+S.+et+al.&rft.au=di+Vimarcati%2C+S.D.C.%3B+Foresti%2C+S.%3B+Jajodia%2C+S.+et+al.&rft.date=2016&rft.volume=24&rft.issue=3&rft.pages=347%E2%80%9378&rft_id=info:doi\/10.3233%2FJCS-160545&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-diVimercatiInteg13-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-diVimercatiInteg13_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-diVimercatiInteg13_17-1\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-diVimercatiInteg13_17-2\">17.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">di Vimarcati, S.D.C.; Foresti, S.; Jajodia, S. et al. (2013). \"Integrity for join queries in the cloud\". <i>IEEE Transactions on Cloud Computing<\/i> <b>1<\/b> (2): 187\u2013200. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTCC.2013.18\" target=\"_blank\">10.1109\/TCC.2013.18<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integrity+for+join+queries+in+the+cloud&rft.jtitle=IEEE+Transactions+on+Cloud+Computing&rft.aulast=di+Vimarcati%2C+S.D.C.%3B+Foresti%2C+S.%3B+Jajodia%2C+S.+et+al.&rft.au=di+Vimarcati%2C+S.D.C.%3B+Foresti%2C+S.%3B+Jajodia%2C+S.+et+al.&rft.date=2013&rft.volume=1&rft.issue=2&rft.pages=187%E2%80%93200&rft_id=info:doi\/10.1109%2FTCC.2013.18&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HudicData13-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HudicData13_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hudic, A.; Islam, S.; Kieseberg, P. et al. (2013). \"Data confidentiality using fragmentation in cloud computing\". <i>International Journal of Pervasive Computing and Communications<\/i> <b>9<\/b> (1): 37\u201351. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1108%2F17427371311315743\" target=\"_blank\">10.1108\/17427371311315743<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+confidentiality+using+fragmentation+in+cloud+computing&rft.jtitle=International+Journal+of+Pervasive+Computing+and+Communications&rft.aulast=Hudic%2C+A.%3B+Islam%2C+S.%3B+Kieseberg%2C+P.+et+al.&rft.au=Hudic%2C+A.%3B+Islam%2C+S.%3B+Kieseberg%2C+P.+et+al.&rft.date=2013&rft.volume=9&rft.issue=1&rft.pages=37%E2%80%9351&rft_id=info:doi\/10.1108%2F17427371311315743&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CirianiEnforcing11-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CirianiEnforcing11_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ciriani, V.; di Vimercati, S.D.C.; Foresti, S. et al. (2011). \"Enforcing Confidentiality and Data Visibility Constraints: An OBDD Approach\". <i>Proceeding of DBSec 2011: Data and Applications Security and Privacy XXV<\/i>: 44\u201359. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-22348-8_6\" target=\"_blank\">10.1007\/978-3-642-22348-8_6<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enforcing+Confidentiality+and+Data+Visibility+Constraints%3A+An+OBDD+Approach&rft.jtitle=Proceeding+of+DBSec+2011%3A+Data+and+Applications+Security+and+Privacy+XXV&rft.aulast=Ciriani%2C+V.%3B+di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.+et+al.&rft.au=Ciriani%2C+V.%3B+di+Vimercati%2C+S.D.C.%3B+Foresti%2C+S.+et+al.&rft.date=2011&rft.pages=44%E2%80%9359&rft_id=info:doi\/10.1007%2F978-3-642-22348-8_6&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FarkasTheInf02-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FarkasTheInf02_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-FarkasTheInf02_20-1\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Farkas, C.; Jajodia, S. (2002). \"The inference problem: A survey\". <i>ACM SIGKDD Explorations Newsletter<\/i> <b>4<\/b> (2): 6\u201311. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F772862.772864\" target=\"_blank\">10.1145\/772862.772864<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+inference+problem%3A+A+survey&rft.jtitle=ACM+SIGKDD+Explorations+Newsletter&rft.aulast=Farkas%2C+C.%3B+Jajodia%2C+S.&rft.au=Farkas%2C+C.%3B+Jajodia%2C+S.&rft.date=2002&rft.volume=4&rft.issue=2&rft.pages=6%E2%80%9311&rft_id=info:doi\/10.1145%2F772862.772864&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuarnieriSecur17-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuarnieriSecur17_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Guarnieri, M.; Marinovic, S.; Basin, D. (2017). \"Securing Databases from Probabilistic Inference\". <i>Proceedings of the 2017 IEEE 30th Computer Security Foundations Symposium<\/i>: 343\u2013359. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCSF.2017.30\" target=\"_blank\">10.1109\/CSF.2017.30<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Securing+Databases+from+Probabilistic+Inference&rft.jtitle=Proceedings+of+the+2017+IEEE+30th+Computer+Security+Foundations+Symposium&rft.aulast=Guarnieri%2C+M.%3B+Marinovic%2C+S.%3B+Basin%2C+D.&rft.au=Guarnieri%2C+M.%3B+Marinovic%2C+S.%3B+Basin%2C+D.&rft.date=2017&rft.pages=343%E2%80%93359&rft_id=info:doi\/10.1109%2FCSF.2017.30&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KatosAFrame11-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KatosAFrame11_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-KatosAFrame11_22-1\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Katos, V.; Vrakas, D.; Katsaros, P. (2011). \"A framework for access control with inference constraints\". <i>Proceedings of the 2011 IEEE 35th Annual Computer Software and Applications Conference<\/i>: 289\u2013297. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCOMPSAC.2011.45\" target=\"_blank\">10.1109\/COMPSAC.2011.45<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+framework+for+access+control+with+inference+constraints&rft.jtitle=Proceedings+of+the+2011+IEEE+35th+Annual+Computer+Software+and+Applications+Conference&rft.aulast=Katos%2C+V.%3B+Vrakas%2C+D.%3B+Katsaros%2C+P.&rft.au=Katos%2C+V.%3B+Vrakas%2C+D.%3B+Katsaros%2C+P.&rft.date=2011&rft.pages=289%E2%80%93297&rft_id=info:doi\/10.1109%2FCOMPSAC.2011.45&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BiskupReducing08-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BiskupReducing08_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Biskup, J.; Embley, D.W.; Lochner, J.-H. (2008). \"Reducing inference control to access control for normalized database schemas\". <i>Information Processing Letters<\/i> <b>106<\/b> (1): 8-12. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ipl.2007.09.007\" target=\"_blank\">10.1016\/j.ipl.2007.09.007<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reducing+inference+control+to+access+control+for+normalized+database+schemas&rft.jtitle=Information+Processing+Letters&rft.aulast=Biskup%2C+J.%3B+Embley%2C+D.W.%3B+Lochner%2C+J.-H.&rft.au=Biskup%2C+J.%3B+Embley%2C+D.W.%3B+Lochner%2C+J.-H.&rft.date=2008&rft.volume=106&rft.issue=1&rft.pages=8-12&rft_id=info:doi\/10.1016%2Fj.ipl.2007.09.007&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BiskupEfficient10-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BiskupEfficient10_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Biskup, J.; Hartmann, S.; Link, S. (2010). \"Efficient Inference Control for Open Relational Queries\". <i>Proceedings of DBSec 2010: Data and Applications Security and Privacy XXIV<\/i>: 162\u201376. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-13739-6_11\" target=\"_blank\">10.1007\/978-3-642-13739-6_11<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Efficient+Inference+Control+for+Open+Relational+Queries&rft.jtitle=Proceedings+of+DBSec+2010%3A+Data+and+Applications+Security+and+Privacy+XXIV&rft.aulast=Biskup%2C+J.%3B+Hartmann%2C+S.%3B+Link%2C+S.&rft.au=Biskup%2C+J.%3B+Hartmann%2C+S.%3B+Link%2C+S.&rft.date=2010&rft.pages=162%E2%80%9376&rft_id=info:doi\/10.1007%2F978-3-642-13739-6_11&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrodskySecure00-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BrodskySecure00_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-BrodskySecure00_25-1\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brodsky, A. (2000). \"Secure databases: Constraints, inference channels, and monitoring disclosures\". <i>IEEE Transactions on Knowledge and Data Engineering<\/i> <b>12<\/b> (6): 900\u201319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2F69.895801\" target=\"_blank\">10.1109\/69.895801<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Secure+databases%3A+Constraints%2C+inference+channels%2C+and+monitoring+disclosures&rft.jtitle=IEEE+Transactions+on+Knowledge+and+Data+Engineering&rft.aulast=Brodsky%2C+A.&rft.au=Brodsky%2C+A.&rft.date=2000&rft.volume=12&rft.issue=6&rft.pages=900%E2%80%9319&rft_id=info:doi\/10.1109%2F69.895801&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChenData06-26\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ChenData06_26-0\">26.0<\/a><\/sup> <sup><a href=\"#cite_ref-ChenData06_26-1\">26.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chen, Y.; Chu, W.W. (2006). \"Database Security Protection Via Inference Detection\". <i>Proceedings of ISI 2006: Intelligence and Security Informatics<\/i>: 452\u201358. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F11760146_40\" target=\"_blank\">10.1007\/11760146_40<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Database+Security+Protection+Via+Inference+Detection&rft.jtitle=Proceedings+of+ISI+2006%3A+Intelligence+and+Security+Informatics&rft.aulast=Chen%2C+Y.%3B+Chu%2C+W.W.&rft.au=Chen%2C+Y.%3B+Chu%2C+W.W.&rft.date=2006&rft.pages=452%E2%80%9358&rft_id=info:doi\/10.1007%2F11760146_40&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SuControl91-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SuControl91_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Su, T.-A.; Ozsoyoglu, G. (1991). \"Controlling FD and MVD inferences in multilevel relational database systems\". <i>IEEE Transactions on Knowledge and Data Engineering<\/i> <b>3<\/b> (4): 474-485. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2F69.109108\" target=\"_blank\">10.1109\/69.109108<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Controlling+FD+and+MVD+inferences+in+multilevel+relational+database+systems&rft.jtitle=IEEE+Transactions+on+Knowledge+and+Data+Engineering&rft.aulast=Su%2C+T.-A.%3B+Ozsoyoglu%2C+G.&rft.au=Su%2C+T.-A.%3B+Ozsoyoglu%2C+G.&rft.date=1991&rft.volume=3&rft.issue=4&rft.pages=474-485&rft_id=info:doi\/10.1109%2F69.109108&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YipData98-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YipData98_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yip, R.W.; Levitt, E.N. (1998). \"Data level inference detection in database systems\". <i>Proceedings of the 11th IEEE Computer Security Foundations Workshop<\/i>: 179\u201389. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCSFW.1998.683168\" target=\"_blank\">10.1109\/CSFW.1998.683168<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+level+inference+detection+in+database+systems&rft.jtitle=Proceedings+of+the+11th+IEEE+Computer+Security+Foundations+Workshop&rft.aulast=Yip%2C+R.W.%3B+Levitt%2C+E.N.&rft.au=Yip%2C+R.W.%3B+Levitt%2C+E.N.&rft.date=1998&rft.pages=179%E2%80%9389&rft_id=info:doi\/10.1109%2FCSFW.1998.683168&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DelugachWizard96-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DelugachWizard96_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Delugach, H.S.; Hinke, T.H. (1996). \"Wizard: A database inference analysis and detection system\". <i>IEEE Transactions on Knowledge and Data Engineering<\/i> <b>8<\/b> (1): 56\u201366. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2F69.485629\" target=\"_blank\">10.1109\/69.485629<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Wizard%3A+A+database+inference+analysis+and+detection+system&rft.jtitle=IEEE+Transactions+on+Knowledge+and+Data+Engineering&rft.aulast=Delugach%2C+H.S.%3B+Hinke%2C+T.H.&rft.au=Delugach%2C+H.S.%3B+Hinke%2C+T.H.&rft.date=1996&rft.volume=8&rft.issue=1&rft.pages=56%E2%80%9366&rft_id=info:doi\/10.1109%2F69.485629&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HinkeAerie92-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HinkeAerie92_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hinke, T.H.; Delugach, H.S. (1992). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/apps.dtic.mil\/sti\/pdfs\/ADA298828.pdf\" target=\"_blank\">\"AERIE: An inference modeling and detection approach for databases\"<\/a> (PDF). <i>Proceedings of IFIP WG 11.3 Sixth Working Conference on Database Security<\/i>: 187\u2013201<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/apps.dtic.mil\/sti\/pdfs\/ADA298828.pdf\" target=\"_blank\">https:\/\/apps.dtic.mil\/sti\/pdfs\/ADA298828.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=AERIE%3A+An+inference+modeling+and+detection+approach+for+databases&rft.jtitle=Proceedings+of+IFIP+WG+11.3+Sixth+Working+Conference+on+Database+Security&rft.aulast=Hinke%2C+T.H.%3B+Delugach%2C+H.S.&rft.au=Hinke%2C+T.H.%3B+Delugach%2C+H.S.&rft.date=1992&rft.pages=187%E2%80%93201&rft_id=https%3A%2F%2Fapps.dtic.mil%2Fsti%2Fpdfs%2FADA298828.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RathATool96-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RathATool96_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rath, S.; Jones, D.; Hale, J. et al. (1996). \"A Tool for Inference Detection and Knowledge Discovery in Databases\". <i>Proceedings from Database Security IX<\/i>: 317\u2013332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-0-387-34932-9_20\" target=\"_blank\">10.1007\/978-0-387-34932-9_20<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Tool+for+Inference+Detection+and+Knowledge+Discovery+in+Databases&rft.jtitle=Proceedings+from+Database+Security+IX&rft.aulast=Rath%2C+S.%3B+Jones%2C+D.%3B+Hale%2C+J.+et+al.&rft.au=Rath%2C+S.%3B+Jones%2C+D.%3B+Hale%2C+J.+et+al.&rft.date=1996&rft.pages=317%E2%80%93332&rft_id=info:doi\/10.1007%2F978-0-387-34932-9_20&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangResist17-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangResist17_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, J.; Yang, J.; Guo, F. et al. (2017). \"Resist the Database Intrusion Caused by Functional Dependency\". <i>Proceedings of the 2017 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery<\/i>: 54\u201357. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCyberC.2017.11\" target=\"_blank\">10.1109\/CyberC.2017.11<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Resist+the+Database+Intrusion+Caused+by+Functional+Dependency&rft.jtitle=Proceedings+of+the+2017+International+Conference+on+Cyber-Enabled+Distributed+Computing+and+Knowledge+Discovery&rft.aulast=Wang%2C+J.%3B+Yang%2C+J.%3B+Guo%2C+F.+et+al.&rft.au=Wang%2C+J.%3B+Yang%2C+J.%3B+Guo%2C+F.+et+al.&rft.date=2017&rft.pages=54%E2%80%9357&rft_id=info:doi\/10.1109%2FCyberC.2017.11&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AnDynam06-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AnDynam06_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">An, X.; Jutla, D.; Cercone, N. (2006). \"Dynamic inference control in privacy preference enforcement\". <i>Proceedings of the 2006 International Conference on Privacy, Security and Trust<\/i>: 1\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1501434.1501464\" target=\"_blank\">10.1145\/1501434.1501464<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Dynamic+inference+control+in+privacy+preference+enforcement&rft.jtitle=Proceedings+of+the+2006+International+Conference+on+Privacy%2C+Security+and+Trust&rft.aulast=An%2C+X.%3B+Jutla%2C+D.%3B+Cercone%2C+N.&rft.au=An%2C+X.%3B+Jutla%2C+D.%3B+Cercone%2C+N.&rft.date=2006&rft.pages=1%E2%80%9310&rft_id=info:doi\/10.1145%2F1501434.1501464&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ThuraisinghamDesign93-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ThuraisinghamDesign93_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thuraisingham, B.; Ford, W.; Collins, M. et al. (1993). \"Design and implementation of a database inference controller\". <i>Data & Knowledge Engineering<\/i> <b>11<\/b> (3): 271-297. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2F0169-023X%2893%2990025-K\" target=\"_blank\">10.1016\/0169-023X(93)90025-K<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Design+and+implementation+of+a+database+inference+controller&rft.jtitle=Data+%26+Knowledge+Engineering&rft.aulast=Thuraisingham%2C+B.%3B+Ford%2C+W.%3B+Collins%2C+M.+et+al.&rft.au=Thuraisingham%2C+B.%3B+Ford%2C+W.%3B+Collins%2C+M.+et+al.&rft.date=1993&rft.volume=11&rft.issue=3&rft.pages=271-297&rft_id=info:doi\/10.1016%2F0169-023X%2893%2990025-K&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YangNewPara07-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YangNewPara07_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yang, Y.; Li, Y.; Deng, R.H. (2007). \"New Paradigm of Inference Control with Trusted Computing\". <i>Proceedings of DBSec 2007: Data and Applications Security XXI<\/i>: 243\u201358. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-540-73538-0_18\" target=\"_blank\">10.1007\/978-3-540-73538-0_18<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=New+Paradigm+of+Inference+Control+with+Trusted+Computing&rft.jtitle=Proceedings+of+DBSec+2007%3A+Data+and+Applications+Security+XXI&rft.aulast=Yang%2C+Y.%3B+Li%2C+Y.%3B+Deng%2C+R.H.&rft.au=Yang%2C+Y.%3B+Li%2C+Y.%3B+Deng%2C+R.H.&rft.date=2007&rft.pages=243%E2%80%9358&rft_id=info:doi\/10.1007%2F978-3-540-73538-0_18&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StaddonDynamic03-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StaddonDynamic03_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dtaddon, J. (2003). \"Dynamic inference control\". <i>Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery<\/i>: 94\u2013100. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F882082.882103\" target=\"_blank\">10.1145\/882082.882103<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Dynamic+inference+control&rft.jtitle=Proceedings+of+the+8th+ACM+SIGMOD+workshop+on+Research+issues+in+data+mining+and+knowledge+discovery&rft.aulast=Dtaddon%2C+J.&rft.au=Dtaddon%2C+J.&rft.date=2003&rft.pages=94%E2%80%93100&rft_id=info:doi\/10.1145%2F882082.882103&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BiskupOnThe11-37\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BiskupOnThe11_37-0\">37.0<\/a><\/sup> <sup><a href=\"#cite_ref-BiskupOnThe11_37-1\">37.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Biskup, J.; Preu\u00df, M.; Wiese, L. (2011). \"On the Inference-Proofness of Database Fragmentation Satisfying Confidentiality Constraints\". <i>Proceedings of ISC 2011: Information Security<\/i>: 246\u201361. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-24861-0_17\" target=\"_blank\">10.1007\/978-3-642-24861-0_17<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=On+the+Inference-Proofness+of+Database+Fragmentation+Satisfying+Confidentiality+Constraints&rft.jtitle=Proceedings+of+ISC+2011%3A+Information+Security&rft.aulast=Biskup%2C+J.%3B+Preu%C3%9F%2C+M.%3B+Wiese%2C+L.&rft.au=Biskup%2C+J.%3B+Preu%C3%9F%2C+M.%3B+Wiese%2C+L.&rft.date=2011&rft.pages=246%E2%80%9361&rft_id=info:doi\/10.1007%2F978-3-642-24861-0_17&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-daSilvaInfer04-38\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-daSilvaInfer04_38-0\">38.0<\/a><\/sup> <sup><a href=\"#cite_ref-daSilvaInfer04_38-1\">38.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">da Silva, J.C.; Klusch, M.; Lodi, S. et al. (2004). \"Inference attacks in peer-to-peer homogeneous distributed data mining\". <i>Proceedings of the 16th European Conference on Artificial Intelligence<\/i>: 450-54. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5555%2F3000001.3000096\" target=\"_blank\">10.5555\/3000001.3000096<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Inference+attacks+in+peer-to-peer+homogeneous+distributed+data+mining&rft.jtitle=Proceedings+of+the+16th+European+Conference+on+Artificial+Intelligence&rft.aulast=da+Silva%2C+J.C.%3B+Klusch%2C+M.%3B+Lodi%2C+S.+et+al.&rft.au=da+Silva%2C+J.C.%3B+Klusch%2C+M.%3B+Lodi%2C+S.+et+al.&rft.date=2004&rft.pages=450-54&rft_id=info:doi\/10.5555%2F3000001.3000096&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HaddadAccess14-39\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HaddadAccess14_39-0\">39.0<\/a><\/sup> <sup><a href=\"#cite_ref-HaddadAccess14_39-1\">39.1<\/a><\/sup> <sup><a href=\"#cite_ref-HaddadAccess14_39-2\">39.2<\/a><\/sup> <sup><a href=\"#cite_ref-HaddadAccess14_39-3\">39.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Haddad, M.; Stevovic, J.; Chiasera, A. et al. (2014). \"Access Control for Data Integration in Presence of Data Dependencies\". <i>Proceedings of DASFAA 2014: Database Systems for Advanced Applications<\/i>: 203\u201317. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-05813-9_14\" target=\"_blank\">10.1007\/978-3-319-05813-9_14<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Access+Control+for+Data+Integration+in+Presence+of+Data+Dependencies&rft.jtitle=Proceedings+of+DASFAA+2014%3A+Database+Systems+for+Advanced+Applications&rft.aulast=Haddad%2C+M.%3B+Stevovic%2C+J.%3B+Chiasera%2C+A.+et+al.&rft.au=Haddad%2C+M.%3B+Stevovic%2C+J.%3B+Chiasera%2C+A.+et+al.&rft.date=2014&rft.pages=203%E2%80%9317&rft_id=info:doi\/10.1007%2F978-3-319-05813-9_14&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SayahInference15-40\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SayahInference15_40-0\">40.0<\/a><\/sup> <sup><a href=\"#cite_ref-SayahInference15_40-1\">40.1<\/a><\/sup> <sup><a href=\"#cite_ref-SayahInference15_40-2\">40.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sayah, T.; Coquery, E.; Thion, R. et al. (2015). \"Inference Leakage Detection for Authorization Policies over RDF Data\". <i>Proceedings of DBSec 2015: Data and Applications Security and Privacy XXIX<\/i>: 346\u201361. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-20810-7_24\" target=\"_blank\">10.1007\/978-3-319-20810-7_24<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Inference+Leakage+Detection+for+Authorization+Policies+over+RDF+Data&rft.jtitle=Proceedings+of+DBSec+2015%3A+Data+and+Applications+Security+and+Privacy+XXIX&rft.aulast=Sayah%2C+T.%3B+Coquery%2C+E.%3B+Thion%2C+R.+et+al.&rft.au=Sayah%2C+T.%3B+Coquery%2C+E.%3B+Thion%2C+R.+et+al.&rft.date=2015&rft.pages=346%E2%80%9361&rft_id=info:doi\/10.1007%2F978-3-319-20810-7_24&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SellamiInfer15-41\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SellamiInfer15_41-0\">41.0<\/a><\/sup> <sup><a href=\"#cite_ref-SellamiInfer15_41-1\">41.1<\/a><\/sup> <sup><a href=\"#cite_ref-SellamiInfer15_41-2\">41.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sellami, M.; Hacid, M.-S.; Gammoudi, M.M. (2015). \"Inference Control in Data Integration Systems\". <i>Proceedings of OTM 2015: On the Move to Meaningful Internet System<\/i>: 285\u2013302. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-26148-5_17\" target=\"_blank\">10.1007\/978-3-319-26148-5_17<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Inference+Control+in+Data+Integration+Systems&rft.jtitle=Proceedings+of+OTM+2015%3A+On+the+Move+to+Meaningful+Internet+System&rft.aulast=Sellami%2C+M.%3B+Hacid%2C+M.-S.%3B+Gammoudi%2C+M.M.&rft.au=Sellami%2C+M.%3B+Hacid%2C+M.-S.%3B+Gammoudi%2C+M.M.&rft.date=2015&rft.pages=285%E2%80%93302&rft_id=info:doi\/10.1007%2F978-3-319-26148-5_17&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TuranGraph18-42\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TuranGraph18_42-0\">42.0<\/a><\/sup> <sup><a href=\"#cite_ref-TuranGraph18_42-1\">42.1<\/a><\/sup> <sup><a href=\"#cite_ref-TuranGraph18_42-2\">42.2<\/a><\/sup> <sup><a href=\"#cite_ref-TuranGraph18_42-3\">42.3<\/a><\/sup> <sup><a href=\"#cite_ref-TuranGraph18_42-4\">42.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Turan, U.; Toroslu, I.H.; Kantarcioglu, M. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/arxiv.org\/abs\/1803.00497\" target=\"_blank\">\"Graph Based Proactive Secure Decomposition Algorithm for Context Dependent Attribute Based Inference Control Problem\"<\/a>. <i>arXiv<\/i>: 1\u201311<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/arxiv.org\/abs\/1803.00497\" target=\"_blank\">https:\/\/arxiv.org\/abs\/1803.00497<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Graph+Based+Proactive+Secure+Decomposition+Algorithm+for+Context+Dependent+Attribute+Based+Inference+Control+Problem&rft.jtitle=arXiv&rft.aulast=Turan%2C+U.%3B+Toroslu%2C+I.H.%3B+Kantarcioglu%2C+M.&rft.au=Turan%2C+U.%3B+Toroslu%2C+I.H.%3B+Kantarcioglu%2C+M.&rft.date=2018&rft.pages=1%E2%80%9311&rft_id=https%3A%2F%2Farxiv.org%2Fabs%2F1803.00497&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TuranSecure17-43\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TuranSecure17_43-0\">43.0<\/a><\/sup> <sup><a href=\"#cite_ref-TuranSecure17_43-1\">43.1<\/a><\/sup> <sup><a href=\"#cite_ref-TuranSecure17_43-2\">43.2<\/a><\/sup> <sup><a href=\"#cite_ref-TuranSecure17_43-3\">43.3<\/a><\/sup> <sup><a href=\"#cite_ref-TuranSecure17_43-4\">43.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Turan, U.; Torslu, I.H.; Kantarc\u0131o\u011flu, M. (2017). \"Secure logical schema and decomposition algorithm for proactive context dependent attribute based inference control\". <i>Data & Knowledge Engineering<\/i> <b>111<\/b>: 1\u201321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.datak.2017.02.002\" target=\"_blank\">10.1016\/j.datak.2017.02.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Secure+logical+schema+and+decomposition+algorithm+for+proactive+context+dependent+attribute+based+inference+control&rft.jtitle=Data+%26+Knowledge+Engineering&rft.aulast=Turan%2C+U.%3B+Torslu%2C+I.H.%3B+Kantarc%C4%B1o%C4%9Flu%2C+M.&rft.au=Turan%2C+U.%3B+Torslu%2C+I.H.%3B+Kantarc%C4%B1o%C4%9Flu%2C+M.&rft.date=2017&rft.volume=111&rft.pages=1%E2%80%9321&rft_id=info:doi\/10.1016%2Fj.datak.2017.02.002&rfr_id=info:sid\/en.wikipedia.org:Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. The original paper listed references alphabetically; this wiki lists them by order of appearance, by design.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150609\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.874 seconds\nReal time usage: 2.612 seconds\nPreprocessor visited node count: 32397\/1000000\nPreprocessor generated node count: 36211\/1000000\nPost\u2010expand include size: 210585\/2097152 bytes\nTemplate argument size: 77636\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 653.838 1 - -total\n 84.93% 555.288 1 - Template:Reflist\n 75.72% 495.090 43 - Template:Cite_journal\n 72.70% 475.329 43 - Template:Citation\/core\n 5.91% 38.644 42 - Template:Citation\/identifier\n 5.73% 37.441 1 - Template:Infobox_journal_article\n 5.47% 35.792 1 - Template:Infobox\n 3.77% 24.645 80 - Template:Infobox\/row\n 3.50% 22.887 43 - Template:Citation\/make_link\n 2.63% 17.194 85 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12451-0!*!0!!en!5!* and timestamp 20210705150606 and revision id 42190\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions\">https:\/\/www.limswiki.org\/index.php\/Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","2537b4210c2770719a918d28c535508d_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/7\/7a\/Fig1_Jebali_JofInfoTelec2020_5-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/12\/Fig2_Jebali_JofInfoTelec2020_5-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/27\/Fig3_Jebali_JofInfoTelec2020_5-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/2\/23\/Fig4_Jebali_JofInfoTelec2020_5-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/af\/Fig5_Jebali_JofInfoTelec2020_5-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/6\/69\/Fig6_Jebali_JofInfoTelec2020_5-1.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/d\/d9\/Fig7_Jebali_JofInfoTelec2020_5-1.jpg"],"2537b4210c2770719a918d28c535508d_timestamp":1625497566,"5125b8e3cfcdb627ab689919e4e8a4b1_type":"article","5125b8e3cfcdb627ab689919e4e8a4b1_title":"Privacy-preserving healthcare informatics: A review (Chong 2021)","5125b8e3cfcdb627ab689919e4e8a4b1_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Privacy-preserving_healthcare_informatics:_A_review","5125b8e3cfcdb627ab689919e4e8a4b1_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Privacy-preserving healthcare informatics: A review\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nPrivacy-preserving healthcare informatics: A reviewJournal\n \nITM Web of ConferencesAuthor(s)\n \nChong, Kah MengAuthor affiliation(s)\n \nUniversiti Tunku Abdul RahmanPrimary contact\n \nkmchong at utar dot edu dot myYear published\n \n2021Volume and issue\n \n36Article #\n \n04005DOI\n \n10.1051\/itmconf\/20213604005ISSN\n \n2271-2097Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.itm-conferences.org\/articles\/itmconf\/abs\/2021\/01\/itmconf_icmsa2021_04005\/Download\n \nhttps:\/\/www.itm-conferences.org\/articles\/itmconf\/pdf\/2021\/01\/itmconf_icmsa2021_04005.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Privacy threats \n\n3.1 Privacy-preserving data publishing \n3.2 Healthcare data \n3.3 Privacy disclosures \n3.4 Attack models \n3.5 Privacy and utility objective of PPDP \n\n\n4 Privacy models \n\n4.1 Data anonymization \n4.2 Differential privacy \n\n\n5 Conclusion \n6 Acknowledgements \n7 References \n8 Notes \n\n\n\nAbstract \nThe electronic health record (EHR) is the key to an efficient healthcare service delivery system. The publication of healthcare data is highly beneficial to healthcare industries and government institutions to support a variety of medical and census research. However, healthcare data contains sensitive information of patients, and the publication of such data could lead to unintended privacy disclosures. In this paper, we present a comprehensive survey of the state-of-the-art privacy-enhancing methods that ensure a secure healthcare data sharing environment. We focus on the recently proposed schemes based on data anonymization and differential privacy approaches in the protection of healthcare data privacy. We highlight the strengths and limitations of the two approaches and discuss some promising future research directions in this area.\nKeywords: data privacy, data sharing, electronic health record, healthcare informatics, inference\n\nIntroduction \nElectronic health record (EHR) systems are increasingly adopted as an important paradigm in the healthcare industry to collect and store patient data, which includes sensitive information such as demographic data, medical history, diagnosis code, medications, treatment plans, hospitalization records, insurance information, immunization dates, allergies, and laboratory and test results. The availability of such big data has provided unprecedented opportunities to improve the efficiency and quality of healthcare services, particularly in improving patient care outcomes and reducing medical costs. EHR data have been published to allow useful analysis as required by the healthcare industry[1] and government institutions.[2][3] Some key examples may include large-scale statistical analytics (e.g., the study of correlation between diseases), clinical decision making, treatment optimization, clustering (e.g., epidemic control), and census surveys. Driven by the potential of EHR systems, a number of EHR repositories have been established, such as the National Database for Autism Research (NDAR), U.K. Data Service, ClinicalTrials.gov, and UNC Health Care (UNCHC).\nAlthough the publication of EHR data is enormously beneficial, it could lead to unintended privacy disclosures. Many conventional cryptography and security methods have been deployed to primarily protect the security of EHR systems, including access control, authentication, and encryption. However, these technologies do not guarantee privacy preservation of sensitive data. That is, the sensitive information of patient could still be inferred from the published data by an adversary. Various regulations and guidelines have been developed to restrict publishable data types, data usage, and data storage, including the Health Insurance Portability and Accountability Act (HIPAA)[4][5], General Data Protection Regulation (GDPR)[6][7], and Personal Data Protection Act.[8] However, there are several limitations to this regulatory approach. First, a high trust level is required of the data recipient that they follow the rules and regulations provided by the data publisher. Yet, there are adversaries who attempt to attack the published data to reidentify a target victim. Second, sensitive data still might be carelessly published due to human error and fall into the wrong hands, which eventually leads to a breach of individual privacy. As such, regulations and guidelines alone do not provide computational guarantee for preserving the privacy of a patient and thus cannot fully prevent such privacy violations. The need of protecting individual data privacy in a hostile environment, while allowing accurate analysis of patient data, has driven the development of effective privacy models in protecting healthcare data.\nIn this paper, we present the privacy issues in healthcare data publication and elaborate on relevant adversarial attack models. With a focus on data anonymization and differential privacy, we discuss the limitations and strengths of these proposed approaches. Finally, we conclude the paper and highlight future research direction in this area.\n\nPrivacy threats \nIn this section, we first discuss privacy-preserving data publishing (PPDP) and the properties of healthcare data. Then, we present the major privacy disclosures in healthcare data publication and show the relevant attack models. Finally, we present the privacy and utility objective in PPDP.\n\nPrivacy-preserving data publishing \nPrivacy-preserving data publishing (PPDP) provides technical solutions that address the privacy and utility preservation challenges of data sharing scenarios. An overview of PPDP is shown in Figure 1, which includes a general data collection and data publishing scenario. \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Overview of privacy-preserving data publishing (PPDP)\n\n\n\nDuring the data collection phase, data of the record owner (patient) are collected by the data holder (hospital) and stored in an EHR. In the data publishing phase, the data holder releases the collected data to the data recipient (e.g., the public or a third party, e.g., an insurance company or medical research center) for further analysis and data mining. However, some of the data recipients (adversary) are not honest and attempt to obtain more information about the record owner beyond the published data, which includes the identity and sensitive data of the record owner. Hence, PPDP serves as a vital process that sanitizes sensitive information to avoid privacy violations of one or more individuals.\n\nHealthcare data \nTypically, stored healthcare data exists as relational data in tabular form. Each row (tuple) corresponds to one record owner, and each column corresponds to a number of distinct attributes, which can be grouped into the following four categories:\n\n Explicit identifier (ID): a set of attributes such as name, social security number, national IDs, mobile number, and drivers license number that uniquely identifies a record owner\n Quasi-identifier (QID): a set of attributes such as date of birth, gender, address, zip code, and hobby that cannot uniquely identify a record owner but can potentially identify the target if combined with some auxiliary information\n Sensitive attribute (SA): sensitive personal information such as diagnosis codes, genomic information, salary, health condition, insurance information, and relationship status that the record owner intends to keep private from unauthorized parties\n Non-sensitive attribute (NSA): a set of attributes such as cookie IDs, hashed email addresses, and mobile advertising IDs generated from an EHR that do not violate the privacy of the record owner if they are disclosed (Note: all attributes that are not categorized as ID, QID, and SA are classified as NSA.)\nEach attribute can be further classified as a numerical attribute (e.g., age, zip code, and date of birth) and non-numerical attribute (e.g., gender, job, and disease). Table 1 shows an example dataset, in which the name of patients is naively anonymized (by removing the names and social security numbers).\n\n\n\n\n\n\n\nTable 1. An example of different types of attributes in a relational table\n\n\n\nName\n\nQuasi-identifier\n\nSensitive attribute\n\n\nAge\n\nZip code\n\nGender\n\nDisease\n\n\n1\n\n23\n\n96038\n\nMale\n\nDiabetes\n\n\n2\n\n28\n\n96070\n\nFemale\n\nDiabetes\n\n\n3\n\n26\n\n96073\n\nMale\n\nDiabetes\n\n\n4\n\n37\n\n96328\n\nMale\n\nCancer\n\n\n5\n\n33\n\n96319\n\nFemale\n\nMental Illness\n\n\n6\n\n33\n\n96388\n\nFemale\n\nDiabetes\n\n\n7\n\n43\n\n96583\n\nMale\n\nDiabetes\n\n\n8\n\n49\n\n96512\n\nFemale\n\nCancer\n\n\n9\n\n45\n\n96590\n\nMale\n\nCancer\n\n\n\nPrivacy disclosures \nA privacy disclosure is defined as a disclosure of personal information that users intend to keep private from an entity which is not authorized to access or have the information. There are three types of privacy disclosures:\n\n Identity disclosure: Identity disclosure, also known as reidentification, is the major privacy threat in publishing healthcare data. It occurs when the true identity of a targeted victim is revealed by an adversary from the published data. In other words, an individual is reidentified when an adversary is able to map a record in the published data to its corresponding patient with high probability (record linkage). For example, if an adversary possesses the information that A is 43 years old, then A is reidentified as record 7 in Table 1.\n Attribute disclosure: This disclosure occurs when an adversary successfully links a victim to their SA information in the published data with high probability (attribute linkage). This SA information could be an SA value (e.g., \"Disease\" in Table 1) or a range that contains the SA value (e.g., medical cost range).\n Membership disclosure: This disclosure occurs when an adversary successfully infers the existence of a targeted victim in the published data with high probability. For example, the inference of an individual in a COVID-19-positive database poses a privacy threat to the individual.\nAttack models \nPrivacy attacks could be launched by matching a published table containing sensitive information about the target victim with some external resources modelling the background knowledge of the attacker. For a successful attack, an adversary may require the following prior knowledge:\n\n The published table, \ud835\udc7b\u2032: An adversary has access to the published table \ud835\udc47\u2032 (which is often an open resource) and knows that \ud835\udc47 is an anonymized data for some table T.\n QID of a targeted victim: An adversary possesses partial or complete QID values about a target from any external resource and the values are accurate. This assumption is realistic as the QID information is easy to acquire from different sources, including real-life inspection data, external demographic data, and and voter list data.\n Knowledge about the distribution of the SA and NSA in table T : For example, an adversary may possess the information of P (disease=diabetes, age>50) and may utilize this knowledge to make additional inferences about records in the published table \ud835\udc47\u2032.\nGenerally, privacy attacks could be launched due to the linkability properties of the QID. Now we discuss the relevant privacy attack models for identity and attribute disclosure. \n\n Linkage attack: An adversary may reidentify the identity and discover the SA values of a targeted record owner by matching the auxiliary QID values with the published table \ud835\udc47\u2032. For example, imagine Table 1 is published without modification and suppose A possesses the knowledge that B lives in zip code 96038, then A infers that B belongs to record 1 (identity disclosure) and has diabetes (attribute disclosure).[9][10][11][12][13]\n Homogeneity attack: This attack discloses the SA values of a target when there is insufficient homogeneity in the SA. That is, the combination of QID is mapped to one SA value only. For example, suppose A knows that B is 28 years old, which belongs to the first equivalence class (an equivalence class is a cluster of records with the same QID values) in Table 2, below (record 1, 2 and 3). Since these records have the same disease, A infers that B suffers from diabetes.[9][14]\n Background knowledge attack: This attack utilizes logical reasoning and additional knowledge about a target to breach the SA values. For example, suppose A knows that C is 43 years old and lives in the zip code 96583, which belongs to the third equivalence class in Table 2, below (record 7, 8 and 9). Nevertheless, the records show that C may have either diabetes or cancer. Based on A\u2019s background knowledge that C is a person who likes sweet foods, A infers that C is diabetic.[9][14]\n Skewness attack: When the overall distribution of SA in the original data is skewed, SA values can be inferred. The SA values have different degrees of sensitivity. For instance, a victim may not mind being known as diabetic as it is a common (majority) disease. However, one would mind being known to have mental illness. According to Table 3, below, the probability of having mental illness is 33.3%, which is much higher than that of real distribution (11.1% in Table 1). Thus, this imposes a privacy threat, since anyone in the equivalence class that has a 33.3% possibility can be inferred to have mental illness, as compared with 11.1% of the overall distribution.[15]\n Similarity attack: This attack discloses SA values when the semantic relationship of distinct SA values in an equivalence class is close. For example, suppose that an adversary infers the possible salaries of a target victim are 2K, 3K, and 4K. Although the numbers represent distinct salary, they are all categorized in the range [2K,4K]. Hence, an adversary could infer that the target has low salary when the SA values are semantically similar.[14][15]\n\n\n\n\n\n\nTable 2. Published data of Table 1\n\n\n\nNumber\n\nQuasi-identifier\n\nSensitive attribute\n\n\nAge\n\nZip code\n\nGender\n\nDisease\n\n\n1\n\n< 30\n\n960**\n\n*\n\nDiabetes\n\n\n2\n\n< 30\n\n960**\n\n*\n\nDiabetes\n\n\n3\n\n< 30\n\n960**\n\n*\n\nDiabetes\n\n\n4\n\n< 40\n\n963**\n\n*\n\nCancer\n\n\n5\n\n< 40\n\n963**\n\n*\n\nMental Illness\n\n\n6\n\n< 40\n\n963**\n\n*\n\nDiabetes\n\n\n7\n\n< 50\n\n965**\n\n*\n\nDiabetes\n\n\n8\n\n< 50\n\n965**\n\n*\n\nCancer\n\n\n9\n\n< 50\n\n965**\n\n*\n\nCancer\n\n\n\n\n\n\n\n\n\nTable 3. Another version of the published data of Table 1\n\n\n\nNumber\n\nQuasi-identifier\n\nSensitive attribute\n\n\nAge\n\nZip code\n\nGender\n\nDisease\n\n\n1\n\n< 30\n\n960**\n\n*\n\nDiabetes\n\n\n2\n\n< 30\n\n960**\n\n*\n\nMental Illness\n\n\n3\n\n< 30\n\n960**\n\n*\n\nCancer\n\n\n4\n\n< 40\n\n963**\n\n*\n\nCancer\n\n\n5\n\n< 40\n\n963**\n\n*\n\nMental Illness\n\n\n6\n\n< 40\n\n963**\n\n*\n\nDiabetes\n\n\n7\n\n< 50\n\n965**\n\n*\n\nDiabetes\n\n\n8\n\n< 50\n\n965**\n\n*\n\nCancer\n\n\n9\n\n< 50\n\n965**\n\n*\n\nMental Illness\n\n\n\nPrivacy and utility objective of PPDP \nPPDP allows computational guarantees on the prevention of privacy disclosures while maintaining the usefulness of the published data. From the privacy aspect, the identity of patients and their corresponding SA values should be concealed from the public. For instance, it is permissible to disclose the information that there exists diabetic patients in the hospital, but the published data should not disclose which patients have diabetes. Utility preservation is another aspect of PPDP, which emphasizes publishing data that is \u201calmost similar\u201d to the original data. Given that M is an arbitrary data mining process, the output of M(T) and M(\ud835\udc47\u2032) should be almost similar: the difference between M(T) and M(\ud835\udc47\u2032) should be less than a threshold t. In most PPDP scenarios, the data mining process M (the usage of the published data) is unknown at the time of publication. This process M could be a simple census statistic or some specified analysis and data exploration, such as pattern mining, association rules, and data modelling. Privacy and utility are two contradictory aspects: publishing high-utility data implies less privacy protection to the record owner and vice versa.\n\nPrivacy models \nIn this section, we present some well-established privacy models that are used to ensure privacy in healthcare data. Particularly, we focus on data anonymization and differential privacy as two mainstream PPDP technologies that are different in their data publishing mechanisms.\n\nData anonymization \nFigure 2 shows a data publishing scenario in data anonymization. An original database is modified before being published as an anonymized database, which is generated by deploying generalization and suppression on the original database. The anonymized database could be studied in place of the original database. Some common data anonymization models to prevent privacy disclosure include k-anonymity[10][11][12][13][14], l-diversity[9], t-closeness[15], and \ud835\udeff-presence.[16]\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Data anonymization scenario\n\n\n\nk-anonymity was developed to address identity disclosure. It requires that, for one record in the table that has some QID value, there exists at least k-1 other records in the table that have the same QID value. Hence, each record is indistinguishable from at least k-1 other records with respect to the QID value in a k-anonymous table. For example, Table 2 and 3 are 3-anonymous tables. In k-anonymity, any individual cannot be reidentified from the published data with a probability of higher than 1\/k. Other variations of k-anonymity include clustering anonymity[11], distribution-preserving k-anonymity[12], optimization-based k-anonymity[13], \ud835\udf03'-sensitive k-anonymity[14], (X,Y)-anonymity[17], (\u03b1, k)-anonymity[18], LKC-privacy[19], and random k-anonymous[20], which prevents identity disclosure by hiding the record of a target in an equivalence class of records with the same QID values. Although the k-anonymity model protects against identity disclosure, it is vulnerable against attribute disclosure. A homogeneity attack and background knowledge attack is possible by deducing the sensitive attribute values from the published data. To provide protection on the sensitive attribute value, l-diversity and t-closeness have been proposed.\nl-diversity requires every QID group to contain at least l distinct sensitive attribute values. For example, Table 3 is a 3-diverse table where there are at least three distinct sensitive attribute values for every QID group. This method depends on the range of the sensitive attribute values. If the number of distinct sensitive attribute values is lower than the desired privacy parameter l, some fictitious data are added to achieve l-diversity. This further leads to excessive modification and may produce biased results in statistical analysis. In addition, l-diversity does not prevent attribute disclosure when the overall distribution of the sensitive attribute is skewed. A skewness attack and similarity attack are still possible to disclose the SA values in l-diversity. k-anonymity and l-diversity have been combined to propose \ud835\udf0f-safe (l, k)-diversity.[21]\nTo address these vulnerabilities, t-closeness has been proposed, which requires that the distribution of a sensitive attribute in any equivalence class to be close to the distribution of the attribute in the overall table. That is, the distance between the distributions is less than a threshold. This property prevents an adversary from making an accurate estimation of the sensitive attribute values and thus preventing attribute disclosure. However, only SA values are modified, while all the QID values remain unchanged in this model. Hence, it does not prevent identity disclosure. Furthermore, t-closeness deploys a brute-force approach to examine each possible partition of the table to find the optimal solution. This process takes an enormous computation time complexity of 2\ud835\udc42(\ud835\udc5b)\ud835\udc42(\ud835\udc5a).\nTo address membership disclosure, \ud835\udeff-presence has been proposed to limit the confidence level of an adversary in inferring the existence of a targeted victim in the published data to at most \ud835\udeff%.\nThere is a significant amount of precedent for a different parameter value of the privacy models, which could be used as benchmarks for efficient data publishing. However, the choice of the privacy parameter value is flexible and depends on the desired privacy and utility objectives of the data publication, provided that \"an acceptable privacy level\" is guaranteed.\n\nDifferential privacy \nFigure 3 shows a data publishing scenario in differential privacy. Differential privacy[22][23][24][25] involves a query answering process, which a data recipient may send a query to the database and the result of that query is probabilistically indistinguishable regardless of the presence of a record in the database. That is, given two databases that differ in exactly one record, a differentially private mechanism provides two randomized outputs that have almost similar probability distributions. In other words, an adversary could not infer the existence of a targeted victim in the published database with high probability. Randomized noise derived from Laplace distribution is added to the result of the query to achieve privacy.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Differential privacy scenario\n\n\n\nThis is a stronger privacy-enhancing technique that addresses all privacy vulnerabilities in data anonymization approaches and it makes no assumption about the background knowledge of any potential adversary. However, it has some privacy and utility limitations. Firstly, the original data could be estimated with high accuracy from repeated queries. If an adversary performs a series of repeated differential privacy queries (k times) on a published database, then the original data could be disambiguated with high probability. Hence, Laplace noise must be injected k times to guarantee that the published data is invulnerable against k times of such queries. When k is large, the utility of the published data is degraded significantly. In a differentially private database, a maximum of q times queries is allowed to the database. This parameter q is called the privacy budget. The privacy of a database cannot be guaranteed if more than q times queries are made to the database. Thus, the database would stop answering further queries and provide no data utility after q times of queries.\nDifferential privacy preserves utility for low-sensitivity queries such as counting, range, and predicate queries, as the presence or absence of a single record changes the result slightly by one. However, a differentially private database could provide extremely inaccurate results for high-sensitivity queries. Examples of high-sensitivity queries include computation of sum, maximum, minimum, averages, and correlation. Hence, a differentially private database is expected to provide highly biased results for more complex queries, such as computation of variance, skewness, and kurtosis.\n\nConclusion \nAlthough healthcare data provide enormous opportunities to various domains, preserving privacy in healthcare data still poses several unsolved privacy and utility challenges. In this paper, we have provided a general overview of healthcare data publishing problems and discussed the state-of-the-art in data anonymization and differential privacy. We highlighted the practical strengths and limitations of these two privacy-enhancing technologies.\nWith future research, it may be of interest to develop a standardization of privacy protection for privacy policy compliance. Healthcare data holders are required to comply with a number of privacy policies to protect the privacy of a user. This may require the data holders to put systems and processes in place to maintain compliance. However, there is no clear indication of which privacy model and protection level should be adopted. In addition, what constitutes as \"an acceptable privacy level\" is not explicitly or clearly defined in any current privacy laws. Furthermore, it would be of interest to design a privacy model that considers data publication in a distributed and dynamic environment, where there are multiple data holders who publish their data independently to a data pool, with the possibility of data overlapping. The problem, however, is how to anonymize and analyze the aggregated data that consists of anonymized data from each publisher. Furthermore, data are collected and published continuously in a dynamic EHR system (such as wearable healthcare devices). The information contained in profiles could be updated from time to time and required to be reflected in the anonymized data.\n\nAcknowledgements \nNone.\n\nReferences \n\n\n\u2191 Senthilkumar, S.A.; Rai, B.K.; Meshram, A.A. et al. (2018). \"Big Data in Healthcare Management: A Review of Literature\". American Journal of Theoretical and Applied Business 4 (2): 57\u201369. doi:10.11648\/j.ajtab.20180402.14.   \n\n\u2191 Dudeck, M.A.; Horan, T.C.; Peterson, K.D. et al. (2011). \"National Healthcare Safety Network (NHSN) Report, data summary for 2010, device-associated module\". American Journal of Infection Control 39 (10): 798-816. doi:10.1016\/j.ajic.2011.10.001. PMID 22133532.   \n\n\u2191 Powell, K.M.; Li, Q.; Gross, C. et al. (2019). \"Ventilator-Associated Events Reported by U.S. Hospitals to the National Healthcare Safety Network, 2015-2017\". Proceedings of the American Thoracic Society 2019 International Conference. doi:10.1164\/ajrccm-conference.2019.199.1_MeetingAbstracts.A3419.   \n\n\u2191 Cohen, I.G.; Mello, M.M. (2018). \"HIPAA and Protecting Health Information in the 21st Century\". JAMA 320 (3): 231\u201332. doi:10.1001\/jama.2018.5630. PMID 29800120.   \n\n\u2191 Obeng, O.; Paul, S. (2019). \"Understanding HIPAA Compliance Practice in Healthcare Organizations in a Cultural Context\". AMCIS 2019 Proceedings: 1\u20135. https:\/\/aisel.aisnet.org\/amcis2019\/info_security_privacy\/info_security_privacy\/1\/ .   \n\n\u2191 Voigt, P.; von dem Bussche, A. (2017). The EU General Data Protection Regulation (GDPR): A Practical Guide. Springer. ISBN 9783319579580.   \n\n\u2191 Tikkinen-Piri, C.; Rohunen, A.; Markkula, J. (2018). \"EU General Data Protection Regulation: Changes and implications for personal data collecting companies\". Computer Law & Security Review 34 (1): 134\u201353. doi:10.1016\/j.clsr.2017.05.015.   \n\n\u2191 Carey, P. (2018). Data Protection: A Practical Guide to UK and EU Law. Oxford University Press. ISBN 9780198815419.   \n\n\u2191 9.0 9.1 9.2 9.3 Machanavajjhala, A.; Gehrke, J.; Kifer, D. et al. (2006). \"L-diversity: Privacy beyond k-anonymity\". Proceedings of the 22nd International Conference on Data Engineering: 24\u201336. doi:10.1109\/ICDE.2006.1.   \n\n\u2191 10.0 10.1 \">Sweeney, L. (2002). \"k-anonymity: A model for protecting privacy\". International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 10 (5): 557\u2013570. doi:10.1142\/S0218488502001648.   \n\n\u2191 11.0 11.1 11.2 Liu, F.; Li, T. (2018). \"A Clustering K-Anonymity Privacy-Preserving Method for Wearable IoT Devices\". Security and Communication Networks 2018: 4945152. doi:10.1155\/2018\/4945152.   \n\n\u2191 12.0 12.1 12.2 Wei, D.; Ramamurthy, K.N.; Varshney, K.R. (2018). \"Distribution\u2010preserving k\u2010anonymity\". Statistical Analysis and Data Mining 11 (6): 253-270. doi:10.1002\/sam.11374.   \n\n\u2191 13.0 13.1 13.2 Lianf, Y.; Samavi, R. (2020). \"Optimization-based k-anonymity algorithms\". Computers & Security 93: 101753. doi:10.1016\/j.cose.2020.101753.   \n\n\u2191 14.0 14.1 14.2 14.3 14.4 Khan, R.; Tao, X.; Anjum, A. et al. (2020). \"\u03b8-Sensitive k-Anonymity: An Anonymization Model for IoT based Electronic Health Records\". electronics 9 (5): 716. doi:10.3390\/electronics9050716.   \n\n\u2191 15.0 15.1 15.2 Li, N.; Li, T.; Venkatasubramanian, S. (2007). \"t-Closeness: Privacy Beyond k-Anonymity and l-Diversity\". Proceedings of the 2007 IEEE 23rd International Conference on Data Engineering: 106\u201315. doi:10.1109\/ICDE.2007.367856.   \n\n\u2191 Nergiz, M.E.; Atzori, M.; Clifton, C. (2007). \"Hiding the presence of individuals from shared databases\". Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data: 665\u201376. doi:10.1145\/1247480.1247554.   \n\n\u2191 Wamg, K.; Fung, B.C.M. (2006). \"Anonymizing sequential releases\". Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining: 414\u201323. doi:10.1145\/1150402.1150449.   \n\n\u2191 Wong, R.C.-W. (2006). \"(\u03b1, k)-anonymity: An enhanced k-anonymity model for privacy preserving data publishing\". Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining: 754\u201359. doi:10.1145\/1150402.1150499.   \n\n\u2191 Mohammed, N.; Fung, B.C.M.; Hung, P.C.K. et al. (2009). \"Anonymizing healthcare data: A case study on the blood transfusion service\". Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining: 1285\u201394. doi:10.1145\/1557019.1557157.   \n\n\u2191 Song, F.; Ma, T.; Tian, Y. et al. (2019). \"A New Method of Privacy Protection: Random k-Anonymous\". IEEE Access 7: 75434-75445. doi:10.1109\/ACCESS.2019.2919165.   \n\n\u2191 Zhu, H.; Liang, H.-B.; Zhao, L. et al. (2018). \"\u03c4-Safe ( l,k )-Diversity Privacy Model for Sequential Publication With High Utility\". IEEE Access 7: 687\u2013701. doi:10.1109\/ACCESS.2018.2885618.   \n\n\u2191 Dwork, C. (2008). \"Differential Privacy: A Survey of Results\". Proceedings of the 2008 International Conference on Theory and Applications of Models of Computation: 1\u201319. doi:10.1007\/978-3-540-79228-4_1.   \n\n\u2191 Alnemari, S.; Romanowski, C.J.; Raj, R.K. (2017). \"An Adaptive Differential Privacy Algorithm for Range Queries over Healthcare Data\". Proceedings of the 2017 IEEE International Conference on Healthcare Informatics: 397\u2013402. doi:10.1109\/ICHI.2017.49.   \n\n\u2191 Li, H.; Dai, Y.; Lin, X. (2015). \"Efficient e-health data release with consistency guarantee under differential privacy\". Proceedings of the 17th International Conference on E-health Networking, Application & Services: 602\u2013608. doi:10.1109\/HealthCom.2015.7454576.   \n\n\u2191 Guttierrez, O.; Saavedra, J.J.; Zurbaran, M. et al. (2018). \"User-Centered Differential Privacy Mechanisms for Electronic Medical Records\". Proceedings of the 2018 International Carnahan Conference on Security Technology: 1\u20135. doi:10.1109\/CCST.2018.8585555.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation and grammar for readability. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Privacy-preserving_healthcare_informatics:_A_review\">https:\/\/www.limswiki.org\/index.php\/Journal:Privacy-preserving_healthcare_informatics:_A_review<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on data management and sharingLIMSwiki journal articles on health informaticsLIMSwiki journal articles on privacy\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 12 April 2021, at 16:49.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 327 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","5125b8e3cfcdb627ab689919e4e8a4b1_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Privacy-preserving_healthcare_informatics_A_review skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Privacy-preserving healthcare informatics: A review<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>The <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health record<\/a> (EHR) is the key to an efficient healthcare service delivery system. The publication of healthcare data is highly beneficial to healthcare industries and government institutions to support a variety of medical and census research. However, healthcare data contains sensitive <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> of patients, and the publication of such data could lead to unintended <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_privacy\" title=\"Information privacy\" class=\"wiki-link\" data-key=\"185f6d9f874e48914b5789317408f782\">privacy<\/a> disclosures. In this paper, we present a comprehensive survey of the state-of-the-art privacy-enhancing methods that ensure a secure healthcare <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_sharing\" title=\"Data sharing\" class=\"wiki-link\" data-key=\"a99d5fda27f755c693c65864d9286130\">data sharing<\/a> environment. We focus on the recently proposed schemes based on data anonymization and differential privacy approaches in the protection of healthcare data privacy. We highlight the strengths and limitations of the two approaches and discuss some promising future research directions in this area.\n<\/p><p><b>Keywords<\/b>: data privacy, data sharing, electronic health record, healthcare informatics, inference\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">Electronic health record<\/a> (EHR) systems are increasingly adopted as an important paradigm in the healthcare industry to collect and store patient data, which includes sensitive <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> such as demographic data, medical history, diagnosis code, medications, treatment plans, <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospitalization<\/a> records, insurance information, immunization dates, allergies, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> and test results. The availability of such big data has provided unprecedented opportunities to improve the efficiency and quality of healthcare services, particularly in improving patient care outcomes and reducing medical costs. EHR data have been published to allow useful analysis as required by the healthcare industry<sup id=\"rdp-ebb-cite_ref-SenthilkumarBigData18_1-0\" class=\"reference\"><a href=\"#cite_note-SenthilkumarBigData18-1\">[1]<\/a><\/sup> and government institutions.<sup id=\"rdp-ebb-cite_ref-DudeckNational11_2-0\" class=\"reference\"><a href=\"#cite_note-DudeckNational11-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PowellVent19_3-0\" class=\"reference\"><a href=\"#cite_note-PowellVent19-3\">[3]<\/a><\/sup> Some key examples may include large-scale statistical analytics (e.g., the study of correlation between diseases), clinical decision making, treatment optimization, clustering (e.g., control), and census surveys. Driven by the potential of EHR systems, a number of EHR repositories have been established, such as the National Database for Autism Research (NDAR), U.K. Data Service, ClinicalTrials.gov, and UNC Health Care (UNCHC).\n<\/p><p>Although the publication of EHR data is enormously beneficial, it could lead to unintended privacy disclosures. Many conventional cryptography and security methods have been deployed to primarily protect the security of EHR systems, including access control, authentication, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Encryption\" title=\"Encryption\" class=\"wiki-link\" data-key=\"86a503652ed5cc9d8e2b0252a480b5e1\">encryption<\/a>. However, these technologies do not guarantee <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_privacy\" title=\"Information privacy\" class=\"wiki-link\" data-key=\"185f6d9f874e48914b5789317408f782\">privacy<\/a> preservation of sensitive data. That is, the sensitive information of patient could still be inferred from the published data by an adversary. Various regulations and guidelines have been developed to restrict publishable data types, data usage, and data storage, including the <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Insurance_Portability_and_Accountability_Act\" title=\"Health Insurance Portability and Accountability Act\" class=\"wiki-link\" data-key=\"b70673a0117c21576016cb7498867153\">Health Insurance Portability and Accountability Act<\/a> (HIPAA)<sup id=\"rdp-ebb-cite_ref-CohenHIPAA18_4-0\" class=\"reference\"><a href=\"#cite_note-CohenHIPAA18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ObengUnder19_5-0\" class=\"reference\"><a href=\"#cite_note-ObengUnder19-5\">[5]<\/a><\/sup>, <a href=\"https:\/\/www.limswiki.org\/index.php\/General_Data_Protection_Regulation\" title=\"General Data Protection Regulation\" class=\"wiki-link\" data-key=\"3f4bdf6f0dcb360b1e79aad8674c2447\">General Data Protection Regulation<\/a> (GDPR)<sup id=\"rdp-ebb-cite_ref-VoigtTheEU17_6-0\" class=\"reference\"><a href=\"#cite_note-VoigtTheEU17-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Tikkinen-PiriEUGen18_7-0\" class=\"reference\"><a href=\"#cite_note-Tikkinen-PiriEUGen18-7\">[7]<\/a><\/sup>, and Personal Data Protection Act.<sup id=\"rdp-ebb-cite_ref-CareyData18_8-0\" class=\"reference\"><a href=\"#cite_note-CareyData18-8\">[8]<\/a><\/sup> However, there are several limitations to this regulatory approach. First, a high trust level is required of the data recipient that they follow the rules and regulations provided by the data publisher. Yet, there are adversaries who attempt to attack the published data to reidentify a target victim. Second, sensitive data still might be carelessly published due to human error and fall into the wrong hands, which eventually leads to a breach of individual privacy. As such, regulations and guidelines alone do not provide computational guarantee for preserving the privacy of a patient and thus cannot fully prevent such privacy violations. The need of protecting individual data privacy in a hostile environment, while allowing accurate analysis of patient data, has driven the development of effective privacy models in protecting healthcare data.\n<\/p><p>In this paper, we present the privacy issues in healthcare data publication and elaborate on relevant adversarial attack models. With a focus on data anonymization and differential privacy, we discuss the limitations and strengths of these proposed approaches. Finally, we conclude the paper and highlight future research direction in this area.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Privacy_threats\">Privacy threats<\/span><\/h2>\n<p>In this section, we first discuss privacy-preserving data publishing (PPDP) and the properties of healthcare data. Then, we present the major privacy disclosures in healthcare data publication and show the relevant attack models. Finally, we present the privacy and utility objective in PPDP.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Privacy-preserving_data_publishing\">Privacy-preserving data publishing<\/span><\/h3>\n<p>Privacy-preserving data publishing (PPDP) provides technical solutions that address the privacy and utility preservation challenges of <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_sharing\" title=\"Data sharing\" class=\"wiki-link\" data-key=\"a99d5fda27f755c693c65864d9286130\">data sharing<\/a> scenarios. An overview of PPDP is shown in Figure 1, which includes a general data collection and data publishing scenario. \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Chong_ITMWebConf21_36.png\" class=\"image wiki-link\" data-key=\"6fc9fdc0d4ac2b5502a3f264dae4a906\"><img alt=\"Fig1 Chong ITMWebConf21 36.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b6\/Fig1_Chong_ITMWebConf21_36.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Overview of privacy-preserving data publishing (PPDP)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>During the data collection phase, data of the record owner (patient) are collected by the data holder (hospital) and stored in an EHR. In the data publishing phase, the data holder releases the collected data to the data recipient (e.g., the public or a third party, e.g., an insurance company or medical research center) for further analysis and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_mining\" title=\"Data mining\" class=\"wiki-link\" data-key=\"be09d3680fe1608addedf6f62692ee47\">data mining<\/a>. However, some of the data recipients (adversary) are not honest and attempt to obtain more information about the record owner beyond the published data, which includes the identity and sensitive data of the record owner. Hence, PPDP serves as a vital process that sanitizes sensitive information to avoid privacy violations of one or more individuals.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Healthcare_data\">Healthcare data<\/span><\/h3>\n<p>Typically, stored healthcare data exists as relational data in tabular form. Each row (tuple) corresponds to one record owner, and each column corresponds to a number of distinct attributes, which can be grouped into the following four categories:\n<\/p>\n<ul><li> <b>Explicit identifier (ID)<\/b>: a set of attributes such as name, social security number, national IDs, mobile number, and drivers license number that uniquely identifies a record owner<\/li>\n<li> <b>Quasi-identifier (QID)<\/b>: a set of attributes such as date of birth, gender, address, zip code, and hobby that cannot uniquely identify a record owner but can potentially identify the target if combined with some auxiliary information<\/li>\n<li> <b>Sensitive attribute (SA)<\/b>: sensitive personal information such as diagnosis codes, genomic information, salary, health condition, insurance information, and relationship status that the record owner intends to keep private from unauthorized parties<\/li>\n<li> <b>Non-sensitive attribute (NSA)<\/b>: a set of attributes such as cookie IDs, hashed email addresses, and mobile advertising IDs generated from an EHR that do not violate the privacy of the record owner if they are disclosed (Note: all attributes that are not categorized as ID, QID, and SA are classified as NSA.)<\/li><\/ul>\n<p>Each attribute can be further classified as a numerical attribute (e.g., age, zip code, and date of birth) and non-numerical attribute (e.g., gender, job, and disease). Table 1 shows an example dataset, in which the name of patients is naively anonymized (by removing the names and social security numbers).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 1.<\/b> An example of different types of attributes in a relational table\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Name\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"3\">Quasi-identifier\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Sensitive attribute\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Age\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Zip code\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Gender\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Disease\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96038\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Male\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96070\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Female\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96073\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Male\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">37\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96328\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Male\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96319\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Female\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mental Illness\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96388\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Female\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">43\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96583\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Male\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">49\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96512\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Female\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">45\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">96590\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Male\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Privacy_disclosures\">Privacy disclosures<\/span><\/h3>\n<p>A privacy disclosure is defined as a disclosure of personal information that users intend to keep private from an entity which is not authorized to access or have the information. There are three types of privacy disclosures:\n<\/p>\n<ul><li> <b>Identity disclosure<\/b>: Identity disclosure, also known as reidentification, is the major privacy threat in publishing healthcare data. It occurs when the true identity of a targeted victim is revealed by an adversary from the published data. In other words, an individual is reidentified when an adversary is able to map a record in the published data to its corresponding patient with high probability (record linkage). For example, if an adversary possesses the information that A is 43 years old, then A is reidentified as record 7 in Table 1.<\/li>\n<li> <b>Attribute disclosure<\/b>: This disclosure occurs when an adversary successfully links a victim to their SA information in the published data with high probability (attribute linkage). This SA information could be an SA value (e.g., \"Disease\" in Table 1) or a range that contains the SA value (e.g., medical cost range).<\/li>\n<li> <b>Membership disclosure<\/b>: This disclosure occurs when an adversary successfully infers the existence of a targeted victim in the published data with high probability. For example, the inference of an individual in a <a href=\"https:\/\/www.limswiki.org\/index.php\/COVID-19\" title=\"COVID-19\" class=\"mw-redirect wiki-link\" data-key=\"da9bd20c492b2a17074ad66c2fe25652\">COVID-19<\/a>-positive database poses a privacy threat to the individual.<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Attack_models\">Attack models<\/span><\/h3>\n<p>Privacy attacks could be launched by matching a published table containing sensitive information about the target victim with some external resources modelling the background knowledge of the attacker. For a successful attack, an adversary may require the following prior knowledge:\n<\/p>\n<ul><li> <b>The published table, \ud835\udc7b\u2032<\/b>: An adversary has access to the published table \ud835\udc47\u2032 (which is often an open resource) and knows that \ud835\udc47 is an anonymized data for some table T.<\/li>\n<li> <b>QID of a targeted victim<\/b>: An adversary possesses partial or complete QID values about a target from any external resource and the values are accurate. This assumption is realistic as the QID information is easy to acquire from different sources, including real-life inspection data, external demographic data, and and voter list data.<\/li>\n<li> <b>Knowledge about the distribution of the SA and NSA in table T<\/b> : For example, an adversary may possess the information of <i>P<\/i> (disease=diabetes, age>50) and may utilize this knowledge to make additional inferences about records in the published table \ud835\udc47\u2032.<\/li><\/ul>\n<p>Generally, privacy attacks could be launched due to the linkability properties of the QID. Now we discuss the relevant privacy attack models for identity and attribute disclosure. \n<\/p>\n<ul><li> <b>Linkage attack<\/b>: An adversary may reidentify the identity and discover the SA values of a targeted record owner by matching the auxiliary QID values with the published table \ud835\udc47\u2032. For example, imagine Table 1 is published without modification and suppose A possesses the knowledge that B lives in zip code 96038, then A infers that B belongs to record 1 (identity disclosure) and has diabetes (attribute disclosure).<sup id=\"rdp-ebb-cite_ref-MachanavajjhalaL-diversity06_9-0\" class=\"reference\"><a href=\"#cite_note-MachanavajjhalaL-diversity06-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Sweeney_k-anon02_10-0\" class=\"reference\"><a href=\"#cite_note-Sweeney_k-anon02-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiuAClust18_11-0\" class=\"reference\"><a href=\"#cite_note-LiuAClust18-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WeiDistrib18_12-0\" class=\"reference\"><a href=\"#cite_note-WeiDistrib18-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiangOptim20_13-0\" class=\"reference\"><a href=\"#cite_note-LiangOptim20-13\">[13]<\/a><\/sup><\/li>\n<li> <b>Homogeneity attack<\/b>: This attack discloses the SA values of a target when there is insufficient homogeneity in the SA. That is, the combination of QID is mapped to one SA value only. For example, suppose A knows that B is 28 years old, which belongs to the first equivalence class (an equivalence class is a cluster of records with the same QID values) in Table 2, below (record 1, 2 and 3). Since these records have the same disease, A infers that B suffers from diabetes.<sup id=\"rdp-ebb-cite_ref-MachanavajjhalaL-diversity06_9-1\" class=\"reference\"><a href=\"#cite_note-MachanavajjhalaL-diversity06-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Khan.CE.B8-Sensitive20_14-0\" class=\"reference\"><a href=\"#cite_note-Khan.CE.B8-Sensitive20-14\">[14]<\/a><\/sup><\/li>\n<li> <b>Background knowledge attack<\/b>: This attack utilizes logical reasoning and additional knowledge about a target to breach the SA values. For example, suppose A knows that C is 43 years old and lives in the zip code 96583, which belongs to the third equivalence class in Table 2, below (record 7, 8 and 9). Nevertheless, the records show that C may have either diabetes or cancer. Based on A\u2019s background knowledge that C is a person who likes sweet foods, A infers that C is diabetic.<sup id=\"rdp-ebb-cite_ref-MachanavajjhalaL-diversity06_9-2\" class=\"reference\"><a href=\"#cite_note-MachanavajjhalaL-diversity06-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Khan.CE.B8-Sensitive20_14-1\" class=\"reference\"><a href=\"#cite_note-Khan.CE.B8-Sensitive20-14\">[14]<\/a><\/sup><\/li>\n<li> <b>Skewness attack<\/b>: When the overall distribution of SA in the original data is skewed, SA values can be inferred. The SA values have different degrees of sensitivity. For instance, a victim may not mind being known as diabetic as it is a common (majority) disease. However, one would mind being known to have mental illness. According to Table 3, below, the probability of having mental illness is 33.3%, which is much higher than that of real distribution (11.1% in Table 1). Thus, this imposes a privacy threat, since anyone in the equivalence class that has a 33.3% possibility can be inferred to have mental illness, as compared with 11.1% of the overall distribution.<sup id=\"rdp-ebb-cite_ref-Lit-Close07_15-0\" class=\"reference\"><a href=\"#cite_note-Lit-Close07-15\">[15]<\/a><\/sup><\/li>\n<li> <b>Similarity attack<\/b>: This attack discloses SA values when the semantic relationship of distinct SA values in an equivalence class is close. For example, suppose that an adversary infers the possible salaries of a target victim are 2K, 3K, and 4K. Although the numbers represent distinct salary, they are all categorized in the range [2K,4K]. Hence, an adversary could infer that the target has low salary when the SA values are semantically similar.<sup id=\"rdp-ebb-cite_ref-Khan.CE.B8-Sensitive20_14-2\" class=\"reference\"><a href=\"#cite_note-Khan.CE.B8-Sensitive20-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Lit-Close07_15-1\" class=\"reference\"><a href=\"#cite_note-Lit-Close07-15\">[15]<\/a><\/sup><\/li><\/ul>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 2.<\/b> Published data of Table 1\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Number\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"3\">Quasi-identifier\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Sensitive attribute\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Age\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Zip code\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Gender\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Disease\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">960**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">960**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">960**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">963**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">963**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mental Illness\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">963**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">965**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">965**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">965**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 3.<\/b> Another version of the published data of Table 1\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Number\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\" colspan=\"3\">Quasi-identifier\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Sensitive attribute\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Age\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Zip code\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Gender\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Disease\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">960**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">960**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mental Illness\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">960**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">963**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">963**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mental Illness\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">963**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">965**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diabetes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">965**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cancer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">965**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mental Illness\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Privacy_and_utility_objective_of_PPDP\">Privacy and utility objective of PPDP<\/span><\/h3>\n<p>PPDP allows computational guarantees on the prevention of privacy disclosures while maintaining the usefulness of the published data. From the privacy aspect, the identity of patients and their corresponding SA values should be concealed from the public. For instance, it is permissible to disclose the information that there exists diabetic patients in the hospital, but the published data should not disclose which patients have diabetes. Utility preservation is another aspect of PPDP, which emphasizes publishing data that is \u201calmost similar\u201d to the original data. Given that <i>M<\/i> is an arbitrary data mining process, the output of <i>M<\/i>(<i>T<\/i>) and <i>M<\/i>(\ud835\udc47\u2032) should be almost similar: the difference between <i>M<\/i>(<i>T<\/i>) and <i>M<\/i>(\ud835\udc47\u2032) should be less than a threshold <i>t<\/i>. In most PPDP scenarios, the data mining process <i>M<\/i> (the usage of the published data) is unknown at the time of publication. This process <i>M<\/i> could be a simple census statistic or some specified analysis and data exploration, such as pattern mining, association rules, and data modelling. Privacy and utility are two contradictory aspects: publishing high-utility data implies less privacy protection to the record owner and vice versa.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Privacy_models\">Privacy models<\/span><\/h2>\n<p>In this section, we present some well-established privacy models that are used to ensure privacy in healthcare data. Particularly, we focus on data anonymization and differential privacy as two mainstream PPDP technologies that are different in their data publishing mechanisms.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_anonymization\">Data anonymization<\/span><\/h3>\n<p>Figure 2 shows a data publishing scenario in data anonymization. An original database is modified before being published as an anonymized database, which is generated by deploying generalization and suppression on the original database. The anonymized database could be studied in place of the original database. Some common data anonymization models to prevent privacy disclosure include <i>k<\/i>-anonymity<sup id=\"rdp-ebb-cite_ref-Sweeney_k-anon02_10-1\" class=\"reference\"><a href=\"#cite_note-Sweeney_k-anon02-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiuAClust18_11-1\" class=\"reference\"><a href=\"#cite_note-LiuAClust18-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WeiDistrib18_12-1\" class=\"reference\"><a href=\"#cite_note-WeiDistrib18-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiangOptim20_13-1\" class=\"reference\"><a href=\"#cite_note-LiangOptim20-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Khan.CE.B8-Sensitive20_14-3\" class=\"reference\"><a href=\"#cite_note-Khan.CE.B8-Sensitive20-14\">[14]<\/a><\/sup>, <i>l<\/i>-diversity<sup id=\"rdp-ebb-cite_ref-MachanavajjhalaL-diversity06_9-3\" class=\"reference\"><a href=\"#cite_note-MachanavajjhalaL-diversity06-9\">[9]<\/a><\/sup>, <i>t<\/i>-closeness<sup id=\"rdp-ebb-cite_ref-Lit-Close07_15-2\" class=\"reference\"><a href=\"#cite_note-Lit-Close07-15\">[15]<\/a><\/sup>, and <i>\ud835\udeff<\/i>-presence.<sup id=\"rdp-ebb-cite_ref-NergizHiding07_16-0\" class=\"reference\"><a href=\"#cite_note-NergizHiding07-16\">[16]<\/a><\/sup>\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Chong_ITMWebConf21_36.png\" class=\"image wiki-link\" data-key=\"0b13f39ada47027fcd4c8cb304676038\"><img alt=\"Fig2 Chong ITMWebConf21 36.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8c\/Fig2_Chong_ITMWebConf21_36.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Data anonymization scenario<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><i>k<\/i>-anonymity was developed to address identity disclosure. It requires that, for one record in the table that has some QID value, there exists at least <i>k<\/i>-1 other records in the table that have the same QID value. Hence, each record is indistinguishable from at least <i>k<\/i>-1 other records with respect to the QID value in a <i>k<\/i>-anonymous table. For example, Table 2 and 3 are 3-anonymous tables. In <i>k<\/i>-anonymity, any individual cannot be reidentified from the published data with a probability of higher than 1\/<i>k<\/i>. Other variations of <i>k<\/i>-anonymity include clustering anonymity<sup id=\"rdp-ebb-cite_ref-LiuAClust18_11-2\" class=\"reference\"><a href=\"#cite_note-LiuAClust18-11\">[11]<\/a><\/sup>, distribution-preserving <i>k<\/i>-anonymity<sup id=\"rdp-ebb-cite_ref-WeiDistrib18_12-2\" class=\"reference\"><a href=\"#cite_note-WeiDistrib18-12\">[12]<\/a><\/sup>, optimization-based <i>k<\/i>-anonymity<sup id=\"rdp-ebb-cite_ref-LiangOptim20_13-2\" class=\"reference\"><a href=\"#cite_note-LiangOptim20-13\">[13]<\/a><\/sup>, <i>\ud835\udf03'<\/i>-sensitive <i>k<\/i>-anonymity<sup id=\"rdp-ebb-cite_ref-Khan.CE.B8-Sensitive20_14-4\" class=\"reference\"><a href=\"#cite_note-Khan.CE.B8-Sensitive20-14\">[14]<\/a><\/sup>, (<i>X<\/i>,<i>Y<\/i>)-anonymity<sup id=\"rdp-ebb-cite_ref-WangAnonym06_17-0\" class=\"reference\"><a href=\"#cite_note-WangAnonym06-17\">[17]<\/a><\/sup>, (<i>\u03b1<\/i>, <i>k<\/i>)-anonymity<sup id=\"rdp-ebb-cite_ref-Wong.CE.B1kanon06_18-0\" class=\"reference\"><a href=\"#cite_note-Wong.CE.B1kanon06-18\">[18]<\/a><\/sup>, <i>LKC<\/i>-privacy<sup id=\"rdp-ebb-cite_ref-MohammedAnony09_19-0\" class=\"reference\"><a href=\"#cite_note-MohammedAnony09-19\">[19]<\/a><\/sup>, and random <i>k<\/i>-anonymous<sup id=\"rdp-ebb-cite_ref-SongANew19_20-0\" class=\"reference\"><a href=\"#cite_note-SongANew19-20\">[20]<\/a><\/sup>, which prevents identity disclosure by hiding the record of a target in an equivalence class of records with the same QID values. Although the <i>k<\/i>-anonymity model protects against identity disclosure, it is vulnerable against attribute disclosure. A homogeneity attack and background knowledge attack is possible by deducing the sensitive attribute values from the published data. To provide protection on the sensitive attribute value, <i>l<\/i>-diversity and <i>t<\/i>-closeness have been proposed.\n<\/p><p><i>l<\/i>-diversity requires every QID group to contain at least <i>l<\/i> distinct sensitive attribute values. For example, Table 3 is a 3-diverse table where there are at least three distinct sensitive attribute values for every QID group. This method depends on the range of the sensitive attribute values. If the number of distinct sensitive attribute values is lower than the desired privacy parameter <i>l<\/i>, some fictitious data are added to achieve <i>l<\/i>-diversity. This further leads to excessive modification and may produce biased results in statistical analysis. In addition, <i>l<\/i>-diversity does not prevent attribute disclosure when the overall distribution of the sensitive attribute is skewed. A skewness attack and similarity attack are still possible to disclose the SA values in <i>l<\/i>-diversity. <i>k<\/i>-anonymity and <i>l<\/i>-diversity have been combined to propose <i>\ud835\udf0f<\/i>-safe (<i>l<\/i>, <i>k<\/i>)-diversity.<sup id=\"rdp-ebb-cite_ref-ZhuSafe18_21-0\" class=\"reference\"><a href=\"#cite_note-ZhuSafe18-21\">[21]<\/a><\/sup>\n<\/p><p>To address these vulnerabilities, <i>t<\/i>-closeness has been proposed, which requires that the distribution of a sensitive attribute in any equivalence class to be close to the distribution of the attribute in the overall table. That is, the distance between the distributions is less than a threshold. This property prevents an adversary from making an accurate estimation of the sensitive attribute values and thus preventing attribute disclosure. However, only SA values are modified, while all the QID values remain unchanged in this model. Hence, it does not prevent identity disclosure. Furthermore, <i>t<\/i>-closeness deploys a brute-force approach to examine each possible partition of the table to find the optimal solution. This process takes an enormous computation time complexity of 2<sup><i>\ud835\udc42<\/i>(<i>\ud835\udc5b<\/i>)<i>\ud835\udc42<\/i>(<i>\ud835\udc5a<\/i>)<\/sup>.\n<\/p><p>To address membership disclosure, <i>\ud835\udeff<\/i>-presence has been proposed to limit the confidence level of an adversary in inferring the existence of a targeted victim in the published data to at most <i>\ud835\udeff<\/i>%.\n<\/p><p>There is a significant amount of precedent for a different parameter value of the privacy models, which could be used as benchmarks for efficient data publishing. However, the choice of the privacy parameter value is flexible and depends on the desired privacy and utility objectives of the data publication, provided that \"an acceptable privacy level\" is guaranteed.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Differential_privacy\">Differential privacy<\/span><\/h3>\n<p>Figure 3 shows a data publishing scenario in differential privacy. Differential privacy<sup id=\"rdp-ebb-cite_ref-DworkDiffer08_22-0\" class=\"reference\"><a href=\"#cite_note-DworkDiffer08-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlnemariAnAdapt17_23-0\" class=\"reference\"><a href=\"#cite_note-AlnemariAnAdapt17-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiEffic15_24-0\" class=\"reference\"><a href=\"#cite_note-LiEffic15-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GuttierezUser18_25-0\" class=\"reference\"><a href=\"#cite_note-GuttierezUser18-25\">[25]<\/a><\/sup> involves a query answering process, which a data recipient may send a query to the database and the result of that query is probabilistically indistinguishable regardless of the presence of a record in the database. That is, given two databases that differ in exactly one record, a differentially private mechanism provides two randomized outputs that have almost similar probability distributions. In other words, an adversary could not infer the existence of a targeted victim in the published database with high probability. Randomized noise derived from Laplace distribution is added to the result of the query to achieve privacy.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Chong_ITMWebConf21_36.png\" class=\"image wiki-link\" data-key=\"8c55c73b61dbe01d96554b9b51ee41c6\"><img alt=\"Fig3 Chong ITMWebConf21 36.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0a\/Fig3_Chong_ITMWebConf21_36.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Differential privacy scenario<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>This is a stronger privacy-enhancing technique that addresses all privacy vulnerabilities in data anonymization approaches and it makes no assumption about the background knowledge of any potential adversary. However, it has some privacy and utility limitations. Firstly, the original data could be estimated with high accuracy from repeated queries. If an adversary performs a series of repeated differential privacy queries (<i>k<\/i> times) on a published database, then the original data could be disambiguated with high probability. Hence, Laplace noise must be injected <i>k<\/i> times to guarantee that the published data is invulnerable against <i>k<\/i> times of such queries. When <i>k<\/i> is large, the utility of the published data is degraded significantly. In a differentially private database, a maximum of <i>q<\/i> times queries is allowed to the database. This parameter <i>q<\/i> is called the privacy budget. The privacy of a database cannot be guaranteed if more than <i>q<\/i> times queries are made to the database. Thus, the database would stop answering further queries and provide no data utility after <i>q<\/i> times of queries.\n<\/p><p>Differential privacy preserves utility for low-sensitivity queries such as counting, range, and predicate queries, as the presence or absence of a single record changes the result slightly by one. However, a differentially private database could provide extremely inaccurate results for high-sensitivity queries. Examples of high-sensitivity queries include computation of sum, maximum, minimum, averages, and correlation. Hence, a differentially private database is expected to provide highly biased results for more complex queries, such as computation of variance, skewness, and kurtosis.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>Although healthcare data provide enormous opportunities to various domains, preserving privacy in healthcare data still poses several unsolved privacy and utility challenges. In this paper, we have provided a general overview of healthcare data publishing problems and discussed the state-of-the-art in data anonymization and differential privacy. We highlighted the practical strengths and limitations of these two privacy-enhancing technologies.\n<\/p><p>With future research, it may be of interest to develop a standardization of privacy protection for privacy policy compliance. Healthcare data holders are required to comply with a number of privacy policies to protect the privacy of a user. This may require the data holders to put systems and processes in place to maintain compliance. However, there is no clear indication of which privacy model and protection level should be adopted. In addition, what constitutes as \"an acceptable privacy level\" is not explicitly or clearly defined in any current privacy laws. Furthermore, it would be of interest to design a privacy model that considers data publication in a distributed and dynamic environment, where there are multiple data holders who publish their data independently to a data pool, with the possibility of data overlapping. The problem, however, is how to anonymize and analyze the aggregated data that consists of anonymized data from each publisher. Furthermore, data are collected and published continuously in a dynamic EHR system (such as wearable healthcare devices). The information contained in profiles could be updated from time to time and required to be reflected in the anonymized data.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>None.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-SenthilkumarBigData18-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SenthilkumarBigData18_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Senthilkumar, S.A.; Rai, B.K.; Meshram, A.A. et al. (2018). \"Big Data in Healthcare Management: A Review of Literature\". <i>American Journal of Theoretical and Applied Business<\/i> <b>4<\/b> (2): 57\u201369. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.11648%2Fj.ajtab.20180402.14\" target=\"_blank\">10.11648\/j.ajtab.20180402.14<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Big+Data+in+Healthcare+Management%3A+A+Review+of+Literature&rft.jtitle=American+Journal+of+Theoretical+and+Applied+Business&rft.aulast=Senthilkumar%2C+S.A.%3B+Rai%2C+B.K.%3B+Meshram%2C+A.A.+et+al.&rft.au=Senthilkumar%2C+S.A.%3B+Rai%2C+B.K.%3B+Meshram%2C+A.A.+et+al.&rft.date=2018&rft.volume=4&rft.issue=2&rft.pages=57%E2%80%9369&rft_id=info:doi\/10.11648%2Fj.ajtab.20180402.14&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DudeckNational11-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DudeckNational11_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dudeck, M.A.; Horan, T.C.; Peterson, K.D. et al. (2011). \"National Healthcare Safety Network (NHSN) Report, data summary for 2010, device-associated module\". <i>American Journal of Infection Control<\/i> <b>39<\/b> (10): 798-816. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ajic.2011.10.001\" target=\"_blank\">10.1016\/j.ajic.2011.10.001<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22133532\" target=\"_blank\">22133532<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=National+Healthcare+Safety+Network+%28NHSN%29+Report%2C+data+summary+for+2010%2C+device-associated+module&rft.jtitle=American+Journal+of+Infection+Control&rft.aulast=Dudeck%2C+M.A.%3B+Horan%2C+T.C.%3B+Peterson%2C+K.D.+et+al.&rft.au=Dudeck%2C+M.A.%3B+Horan%2C+T.C.%3B+Peterson%2C+K.D.+et+al.&rft.date=2011&rft.volume=39&rft.issue=10&rft.pages=798-816&rft_id=info:doi\/10.1016%2Fj.ajic.2011.10.001&rft_id=info:pmid\/22133532&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PowellVent19-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PowellVent19_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Powell, K.M.; Li, Q.; Gross, C. et al. (2019). \"Ventilator-Associated Events Reported by U.S. Hospitals to the National Healthcare Safety Network, 2015-2017\". <i>Proceedings of the American Thoracic Society 2019 International Conference<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1164%2Fajrccm-conference.2019.199.1_MeetingAbstracts.A3419\" target=\"_blank\">10.1164\/ajrccm-conference.2019.199.1_MeetingAbstracts.A3419<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ventilator-Associated+Events+Reported+by+U.S.+Hospitals+to+the+National+Healthcare+Safety+Network%2C+2015-2017&rft.jtitle=Proceedings+of+the+American+Thoracic+Society+2019+International+Conference&rft.aulast=Powell%2C+K.M.%3B+Li%2C+Q.%3B+Gross%2C+C.+et+al.&rft.au=Powell%2C+K.M.%3B+Li%2C+Q.%3B+Gross%2C+C.+et+al.&rft.date=2019&rft_id=info:doi\/10.1164%2Fajrccm-conference.2019.199.1_MeetingAbstracts.A3419&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CohenHIPAA18-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CohenHIPAA18_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cohen, I.G.; Mello, M.M. (2018). \"HIPAA and Protecting Health Information in the 21st Century\". <i>JAMA<\/i> <b>320<\/b> (3): 231\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2018.5630\" target=\"_blank\">10.1001\/jama.2018.5630<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29800120\" target=\"_blank\">29800120<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=HIPAA+and+Protecting+Health+Information+in+the+21st+Century&rft.jtitle=JAMA&rft.aulast=Cohen%2C+I.G.%3B+Mello%2C+M.M.&rft.au=Cohen%2C+I.G.%3B+Mello%2C+M.M.&rft.date=2018&rft.volume=320&rft.issue=3&rft.pages=231%E2%80%9332&rft_id=info:doi\/10.1001%2Fjama.2018.5630&rft_id=info:pmid\/29800120&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ObengUnder19-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ObengUnder19_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Obeng, O.; Paul, S. (2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/aisel.aisnet.org\/amcis2019\/info_security_privacy\/info_security_privacy\/1\/\" target=\"_blank\">\"Understanding HIPAA Compliance Practice in Healthcare Organizations in a Cultural Context\"<\/a>. <i>AMCIS 2019 Proceedings<\/i>: 1\u20135<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/aisel.aisnet.org\/amcis2019\/info_security_privacy\/info_security_privacy\/1\/\" target=\"_blank\">https:\/\/aisel.aisnet.org\/amcis2019\/info_security_privacy\/info_security_privacy\/1\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+HIPAA+Compliance+Practice+in+Healthcare+Organizations+in+a+Cultural+Context&rft.jtitle=AMCIS+2019+Proceedings&rft.aulast=Obeng%2C+O.%3B+Paul%2C+S.&rft.au=Obeng%2C+O.%3B+Paul%2C+S.&rft.date=2019&rft.pages=1%E2%80%935&rft_id=https%3A%2F%2Faisel.aisnet.org%2Famcis2019%2Finfo_security_privacy%2Finfo_security_privacy%2F1%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VoigtTheEU17-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VoigtTheEU17_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Voigt, P.; von dem Bussche, A. (2017). <i>The EU General Data Protection Regulation (GDPR): A Practical Guide<\/i>. Springer. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783319579580.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+EU+General+Data+Protection+Regulation+%28GDPR%29%3A+A+Practical+Guide&rft.aulast=Voigt%2C+P.%3B+von+dem+Bussche%2C+A.&rft.au=Voigt%2C+P.%3B+von+dem+Bussche%2C+A.&rft.date=2017&rft.pub=Springer&rft.isbn=9783319579580&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Tikkinen-PiriEUGen18-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Tikkinen-PiriEUGen18_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tikkinen-Piri, C.; Rohunen, A.; Markkula, J. (2018). \"EU General Data Protection Regulation: Changes and implications for personal data collecting companies\". <i>Computer Law & Security Review<\/i> <b>34<\/b> (1): 134\u201353. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.clsr.2017.05.015\" target=\"_blank\">10.1016\/j.clsr.2017.05.015<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EU+General+Data+Protection+Regulation%3A+Changes+and+implications+for+personal+data+collecting+companies&rft.jtitle=Computer+Law+%26+Security+Review&rft.aulast=Tikkinen-Piri%2C+C.%3B+Rohunen%2C+A.%3B+Markkula%2C+J.&rft.au=Tikkinen-Piri%2C+C.%3B+Rohunen%2C+A.%3B+Markkula%2C+J.&rft.date=2018&rft.volume=34&rft.issue=1&rft.pages=134%E2%80%9353&rft_id=info:doi\/10.1016%2Fj.clsr.2017.05.015&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CareyData18-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CareyData18_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Carey, P. (2018). <i>Data Protection: A Practical Guide to UK and EU Law<\/i>. Oxford University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780198815419.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Data+Protection%3A+A+Practical+Guide+to+UK+and+EU+Law&rft.aulast=Carey%2C+P.&rft.au=Carey%2C+P.&rft.date=2018&rft.pub=Oxford+University+Press&rft.isbn=9780198815419&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MachanavajjhalaL-diversity06-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MachanavajjhalaL-diversity06_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-MachanavajjhalaL-diversity06_9-1\">9.1<\/a><\/sup> <sup><a href=\"#cite_ref-MachanavajjhalaL-diversity06_9-2\">9.2<\/a><\/sup> <sup><a href=\"#cite_ref-MachanavajjhalaL-diversity06_9-3\">9.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Machanavajjhala, A.; Gehrke, J.; Kifer, D. et al. (2006). \"L-diversity: Privacy beyond k-anonymity\". <i>Proceedings of the 22nd International Conference on Data Engineering<\/i>: 24\u201336. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FICDE.2006.1\" target=\"_blank\">10.1109\/ICDE.2006.1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=L-diversity%3A+Privacy+beyond+k-anonymity&rft.jtitle=Proceedings+of+the+22nd+International+Conference+on+Data+Engineering&rft.aulast=Machanavajjhala%2C+A.%3B+Gehrke%2C+J.%3B+Kifer%2C+D.+et+al.&rft.au=Machanavajjhala%2C+A.%3B+Gehrke%2C+J.%3B+Kifer%2C+D.+et+al.&rft.date=2006&rft.pages=24%E2%80%9336&rft_id=info:doi\/10.1109%2FICDE.2006.1&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Sweeney_k-anon02-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Sweeney_k-anon02_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-Sweeney_k-anon02_10-1\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\">\"><span class=\"citation Journal\">Sweeney, L. (2002). \"k-anonymity: A model for protecting privacy\". <i>International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems<\/i> <b>10<\/b> (5): 557\u2013570. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1142%2FS0218488502001648\" target=\"_blank\">10.1142\/S0218488502001648<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=k-anonymity%3A+A+model+for+protecting+privacy&rft.jtitle=International+Journal+of+Uncertainty%2C+Fuzziness+and+Knowledge-Based+Systems&rft.aulast=Sweeney%2C+L.&rft.au=Sweeney%2C+L.&rft.date=2002&rft.volume=10&rft.issue=5&rft.pages=557%E2%80%93570&rft_id=info:doi\/10.1142%2FS0218488502001648&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiuAClust18-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiuAClust18_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiuAClust18_11-1\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-LiuAClust18_11-2\">11.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liu, F.; Li, T. (2018). \"A Clustering K-Anonymity Privacy-Preserving Method for Wearable IoT Devices\". <i>Security and Communication Networks<\/i> <b>2018<\/b>: 4945152. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1155%2F2018%2F4945152\" target=\"_blank\">10.1155\/2018\/4945152<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Clustering+K-Anonymity+Privacy-Preserving+Method+for+Wearable+IoT+Devices&rft.jtitle=Security+and+Communication+Networks&rft.aulast=Liu%2C+F.%3B+Li%2C+T.&rft.au=Liu%2C+F.%3B+Li%2C+T.&rft.date=2018&rft.volume=2018&rft.pages=4945152&rft_id=info:doi\/10.1155%2F2018%2F4945152&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeiDistrib18-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WeiDistrib18_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-WeiDistrib18_12-1\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-WeiDistrib18_12-2\">12.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wei, D.; Ramamurthy, K.N.; Varshney, K.R. (2018). \"Distribution\u2010preserving k\u2010anonymity\". <i>Statistical Analysis and Data Mining<\/i> <b>11<\/b> (6): 253-270. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fsam.11374\" target=\"_blank\">10.1002\/sam.11374<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Distribution%E2%80%90preserving+k%E2%80%90anonymity&rft.jtitle=Statistical+Analysis+and+Data+Mining&rft.aulast=Wei%2C+D.%3B+Ramamurthy%2C+K.N.%3B+Varshney%2C+K.R.&rft.au=Wei%2C+D.%3B+Ramamurthy%2C+K.N.%3B+Varshney%2C+K.R.&rft.date=2018&rft.volume=11&rft.issue=6&rft.pages=253-270&rft_id=info:doi\/10.1002%2Fsam.11374&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiangOptim20-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiangOptim20_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiangOptim20_13-1\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-LiangOptim20_13-2\">13.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lianf, Y.; Samavi, R. (2020). \"Optimization-based k-anonymity algorithms\". <i>Computers & Security<\/i> <b>93<\/b>: 101753. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cose.2020.101753\" target=\"_blank\">10.1016\/j.cose.2020.101753<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Optimization-based+k-anonymity+algorithms&rft.jtitle=Computers+%26+Security&rft.aulast=Lianf%2C+Y.%3B+Samavi%2C+R.&rft.au=Lianf%2C+Y.%3B+Samavi%2C+R.&rft.date=2020&rft.volume=93&rft.pages=101753&rft_id=info:doi\/10.1016%2Fj.cose.2020.101753&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Khan.CE.B8-Sensitive20-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Khan.CE.B8-Sensitive20_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-Khan.CE.B8-Sensitive20_14-1\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-Khan.CE.B8-Sensitive20_14-2\">14.2<\/a><\/sup> <sup><a href=\"#cite_ref-Khan.CE.B8-Sensitive20_14-3\">14.3<\/a><\/sup> <sup><a href=\"#cite_ref-Khan.CE.B8-Sensitive20_14-4\">14.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khan, R.; Tao, X.; Anjum, A. et al. (2020). \"\u03b8-Sensitive k-Anonymity: An Anonymization Model for IoT based Electronic Health Records\". <i>electronics<\/i> <b>9<\/b> (5): 716. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Felectronics9050716\" target=\"_blank\">10.3390\/electronics9050716<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%CE%B8-Sensitive+k-Anonymity%3A+An+Anonymization+Model+for+IoT+based+Electronic+Health+Records&rft.jtitle=electronics&rft.aulast=Khan%2C+R.%3B+Tao%2C+X.%3B+Anjum%2C+A.+et+al.&rft.au=Khan%2C+R.%3B+Tao%2C+X.%3B+Anjum%2C+A.+et+al.&rft.date=2020&rft.volume=9&rft.issue=5&rft.pages=716&rft_id=info:doi\/10.3390%2Felectronics9050716&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Lit-Close07-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Lit-Close07_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-Lit-Close07_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-Lit-Close07_15-2\">15.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, N.; Li, T.; Venkatasubramanian, S. (2007). \"t-Closeness: Privacy Beyond k-Anonymity and l-Diversity\". <i>Proceedings of the 2007 IEEE 23rd International Conference on Data Engineering<\/i>: 106\u201315. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FICDE.2007.367856\" target=\"_blank\">10.1109\/ICDE.2007.367856<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=t-Closeness%3A+Privacy+Beyond+k-Anonymity+and+l-Diversity&rft.jtitle=Proceedings+of+the+2007+IEEE+23rd+International+Conference+on+Data+Engineering&rft.aulast=Li%2C+N.%3B+Li%2C+T.%3B+Venkatasubramanian%2C+S.&rft.au=Li%2C+N.%3B+Li%2C+T.%3B+Venkatasubramanian%2C+S.&rft.date=2007&rft.pages=106%E2%80%9315&rft_id=info:doi\/10.1109%2FICDE.2007.367856&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NergizHiding07-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NergizHiding07_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nergiz, M.E.; Atzori, M.; Clifton, C. (2007). \"Hiding the presence of individuals from shared databases\". <i>Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data<\/i>: 665\u201376. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1247480.1247554\" target=\"_blank\">10.1145\/1247480.1247554<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hiding+the+presence+of+individuals+from+shared+databases&rft.jtitle=Proceedings+of+the+2007+ACM+SIGMOD+International+Conference+on+Management+of+Data&rft.aulast=Nergiz%2C+M.E.%3B+Atzori%2C+M.%3B+Clifton%2C+C.&rft.au=Nergiz%2C+M.E.%3B+Atzori%2C+M.%3B+Clifton%2C+C.&rft.date=2007&rft.pages=665%E2%80%9376&rft_id=info:doi\/10.1145%2F1247480.1247554&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangAnonym06-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangAnonym06_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wamg, K.; Fung, B.C.M. (2006). \"Anonymizing sequential releases\". <i>Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining<\/i>: 414\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1150402.1150449\" target=\"_blank\">10.1145\/1150402.1150449<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Anonymizing+sequential+releases&rft.jtitle=Proceedings+of+the+12th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining&rft.aulast=Wamg%2C+K.%3B+Fung%2C+B.C.M.&rft.au=Wamg%2C+K.%3B+Fung%2C+B.C.M.&rft.date=2006&rft.pages=414%E2%80%9323&rft_id=info:doi\/10.1145%2F1150402.1150449&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Wong.CE.B1kanon06-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Wong.CE.B1kanon06_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wong, R.C.-W. (2006). \"(\u03b1, k)-anonymity: An enhanced k-anonymity model for privacy preserving data publishing\". <i>Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining<\/i>: 754\u201359. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1150402.1150499\" target=\"_blank\">10.1145\/1150402.1150499<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%28%CE%B1%2C+k%29-anonymity%3A+An+enhanced+k-anonymity+model+for+privacy+preserving+data+publishing&rft.jtitle=Proceedings+of+the+12th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining&rft.aulast=Wong%2C+R.C.-W.&rft.au=Wong%2C+R.C.-W.&rft.date=2006&rft.pages=754%E2%80%9359&rft_id=info:doi\/10.1145%2F1150402.1150499&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MohammedAnony09-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MohammedAnony09_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mohammed, N.; Fung, B.C.M.; Hung, P.C.K. et al. (2009). \"Anonymizing healthcare data: A case study on the blood transfusion service\". <i>Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining<\/i>: 1285\u201394. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1557019.1557157\" target=\"_blank\">10.1145\/1557019.1557157<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Anonymizing+healthcare+data%3A+A+case+study+on+the+blood+transfusion+service&rft.jtitle=Proceedings+of+the+15th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining&rft.aulast=Mohammed%2C+N.%3B+Fung%2C+B.C.M.%3B+Hung%2C+P.C.K.+et+al.&rft.au=Mohammed%2C+N.%3B+Fung%2C+B.C.M.%3B+Hung%2C+P.C.K.+et+al.&rft.date=2009&rft.pages=1285%E2%80%9394&rft_id=info:doi\/10.1145%2F1557019.1557157&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SongANew19-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SongANew19_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Song, F.; Ma, T.; Tian, Y. et al. (2019). \"A New Method of Privacy Protection: Random k-Anonymous\". <i>IEEE Access<\/i> <b>7<\/b>: 75434-75445. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2019.2919165\" target=\"_blank\">10.1109\/ACCESS.2019.2919165<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+New+Method+of+Privacy+Protection%3A+Random+k-Anonymous&rft.jtitle=IEEE+Access&rft.aulast=Song%2C+F.%3B+Ma%2C+T.%3B+Tian%2C+Y.+et+al.&rft.au=Song%2C+F.%3B+Ma%2C+T.%3B+Tian%2C+Y.+et+al.&rft.date=2019&rft.volume=7&rft.pages=75434-75445&rft_id=info:doi\/10.1109%2FACCESS.2019.2919165&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhuSafe18-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhuSafe18_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhu, H.; Liang, H.-B.; Zhao, L. et al. (2018). \"\u03c4-Safe ( l,k )-Diversity Privacy Model for Sequential Publication With High Utility\". <i>IEEE Access<\/i> <b>7<\/b>: 687\u2013701. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FACCESS.2018.2885618\" target=\"_blank\">10.1109\/ACCESS.2018.2885618<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%CF%84-Safe+%28+l%2Ck+%29-Diversity+Privacy+Model+for+Sequential+Publication+With+High+Utility&rft.jtitle=IEEE+Access&rft.aulast=Zhu%2C+H.%3B+Liang%2C+H.-B.%3B+Zhao%2C+L.+et+al.&rft.au=Zhu%2C+H.%3B+Liang%2C+H.-B.%3B+Zhao%2C+L.+et+al.&rft.date=2018&rft.volume=7&rft.pages=687%E2%80%93701&rft_id=info:doi\/10.1109%2FACCESS.2018.2885618&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DworkDiffer08-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DworkDiffer08_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dwork, C. (2008). \"Differential Privacy: A Survey of Results\". <i>Proceedings of the 2008 International Conference on Theory and Applications of Models of Computation<\/i>: 1\u201319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-540-79228-4_1\" target=\"_blank\">10.1007\/978-3-540-79228-4_1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Differential+Privacy%3A+A+Survey+of+Results&rft.jtitle=Proceedings+of+the+2008+International+Conference+on+Theory+and+Applications+of+Models+of+Computation&rft.aulast=Dwork%2C+C.&rft.au=Dwork%2C+C.&rft.date=2008&rft.pages=1%E2%80%9319&rft_id=info:doi\/10.1007%2F978-3-540-79228-4_1&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlnemariAnAdapt17-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlnemariAnAdapt17_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alnemari, S.; Romanowski, C.J.; Raj, R.K. (2017). \"An Adaptive Differential Privacy Algorithm for Range Queries over Healthcare Data\". <i>Proceedings of the 2017 IEEE International Conference on Healthcare Informatics<\/i>: 397\u2013402. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FICHI.2017.49\" target=\"_blank\">10.1109\/ICHI.2017.49<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Adaptive+Differential+Privacy+Algorithm+for+Range+Queries+over+Healthcare+Data&rft.jtitle=Proceedings+of+the+2017+IEEE+International+Conference+on+Healthcare+Informatics&rft.aulast=Alnemari%2C+S.%3B+Romanowski%2C+C.J.%3B+Raj%2C+R.K.&rft.au=Alnemari%2C+S.%3B+Romanowski%2C+C.J.%3B+Raj%2C+R.K.&rft.date=2017&rft.pages=397%E2%80%93402&rft_id=info:doi\/10.1109%2FICHI.2017.49&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiEffic15-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiEffic15_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H.; Dai, Y.; Lin, X. (2015). \"Efficient e-health data release with consistency guarantee under differential privacy\". <i>Proceedings of the 17th International Conference on E-health Networking, Application & Services<\/i>: 602\u2013608. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FHealthCom.2015.7454576\" target=\"_blank\">10.1109\/HealthCom.2015.7454576<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Efficient+e-health+data+release+with+consistency+guarantee+under+differential+privacy&rft.jtitle=Proceedings+of+the+17th+International+Conference+on+E-health+Networking%2C+Application+%26+Services&rft.aulast=Li%2C+H.%3B+Dai%2C+Y.%3B+Lin%2C+X.&rft.au=Li%2C+H.%3B+Dai%2C+Y.%3B+Lin%2C+X.&rft.date=2015&rft.pages=602%E2%80%93608&rft_id=info:doi\/10.1109%2FHealthCom.2015.7454576&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuttierezUser18-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuttierezUser18_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Guttierrez, O.; Saavedra, J.J.; Zurbaran, M. et al. (2018). \"User-Centered Differential Privacy Mechanisms for Electronic Medical Records\". <i>Proceedings of the 2018 International Carnahan Conference on Security Technology<\/i>: 1\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCCST.2018.8585555\" target=\"_blank\">10.1109\/CCST.2018.8585555<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=User-Centered+Differential+Privacy+Mechanisms+for+Electronic+Medical+Records&rft.jtitle=Proceedings+of+the+2018+International+Carnahan+Conference+on+Security+Technology&rft.aulast=Guttierrez%2C+O.%3B+Saavedra%2C+J.J.%3B+Zurbaran%2C+M.+et+al.&rft.au=Guttierrez%2C+O.%3B+Saavedra%2C+J.J.%3B+Zurbaran%2C+M.+et+al.&rft.date=2018&rft.pages=1%E2%80%935&rft_id=info:doi\/10.1109%2FCCST.2018.8585555&rfr_id=info:sid\/en.wikipedia.org:Journal:Privacy-preserving_healthcare_informatics:_A_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation and grammar for readability. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150606\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.504 seconds\nReal time usage: 1.332 seconds\nPreprocessor visited node count: 19552\/1000000\nPreprocessor generated node count: 33880\/1000000\nPost\u2010expand include size: 123118\/2097152 bytes\nTemplate argument size: 44177\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 400.925 1 - -total\n 83.01% 332.791 1 - Template:Reflist\n 70.18% 281.379 25 - Template:Citation\/core\n 67.77% 271.713 23 - Template:Cite_journal\n 9.46% 37.933 1 - Template:Infobox_journal_article\n 9.01% 36.139 1 - Template:Infobox\n 6.18% 24.770 26 - Template:Citation\/identifier\n 6.15% 24.675 80 - Template:Infobox\/row\n 6.13% 24.572 2 - Template:Cite_book\n 3.23% 12.956 25 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12460-0!*!0!!en!5!* and timestamp 20210705150605 and revision id 42237\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Privacy-preserving_healthcare_informatics:_A_review\">https:\/\/www.limswiki.org\/index.php\/Journal:Privacy-preserving_healthcare_informatics:_A_review<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","5125b8e3cfcdb627ab689919e4e8a4b1_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/b\/b6\/Fig1_Chong_ITMWebConf21_36.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8c\/Fig2_Chong_ITMWebConf21_36.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/0a\/Fig3_Chong_ITMWebConf21_36.png"],"5125b8e3cfcdb627ab689919e4e8a4b1_timestamp":1625497564,"e49209f2666c8e463be219dc620f0ec5_type":"article","e49209f2666c8e463be219dc620f0ec5_title":"Blockchain-based healthcare workflow for IoT-connected laboratories in federated hospital clouds (Celesti et al. 2020)","e49209f2666c8e463be219dc620f0ec5_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds","e49209f2666c8e463be219dc620f0ec5_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Blockchain-based healthcare workflow for IoT-connected laboratories in federated hospital clouds\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nBlockchain-based healthcare workflow for IoT-connected laboratories in federated hospital cloudsJournal\n \nSensorsAuthor(s)\n \nCelesti, Antonio; Ruggeri, Armando; Fazio, Maria; Galletta, Antonino; Villari, Massimo; Romano, AgataAuthor affiliation(s)\n \nUniversity of Messina, INdAM-GNCS, IRCCS Centro Neurolesi Bonino Pulejo, ASP MessinaPrimary contact\n \nEmail: acelesti at unime dot itYear published\n \n2020Volume and issue\n \n20(9)Article #\n \n2590DOI\n \n10.3390\/s20092590ISSN\n \n1424-8220Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/1424-8220\/20\/9\/2590\/htmDownload\n \nhttps:\/\/www.mdpi.com\/1424-8220\/20\/9\/2590\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Related work \n4 Motivation \n\n4.1 Recent advancements in diagnostic laboratory devices \n4.2 Towards IoT-connected laboratories \n\n\n5 System design \n6 System prototype \n7 Experiments \n8 Conclusions and future work \n9 Acknowledgements \n\n9.1 Author contributions \n9.2 Funding \n9.3 Conflict of interest \n\n\n10 References \n11 Notes \n\n\n\nAbstract \nIn a pandemic-related situation such as that caused by the SARS-CoV-2 virus, the need for telemedicine and other distanced services becomes dramatically fundamental to reducing the movement of patients, and by extension reducing the risk of infection in healthcare settings. One potential avenue for achieving this is through leveraging cloud computing and internet of things (IoT) technologies. This paper proposes an IoT-connected laboratory service where clinical exams are performed on patients directly in a hospital by technicians through the use of IoT medical diagnostic devices, with results automatically being sent via the hospital's cloud to doctors of federated hospitals for validation and\/or consultation. In particular, we discuss a distributed scenario where nurses, technicians, and medical doctors belonging to different hospitals cooperate through their federated hospital clouds to form a virtual health team able to carry out a healthcare workflow in secure fashion by leveraging the intrinsic security features of blockchain technology. In particular, both public and hybrid blockchain scenarios are discussed and assessed using the Ethereum platform.\nKeywords: blockchain, smart contract, workflow, healthcare, hospital, cloud computing, IoT, data federation\n\nIntroduction \nRecent advancements in information and communication technology have paved the way toward new innovative telemedicine and other remote healthcare services, which are able to support the growing demand of even more accessible medical treatments.[1][2] Particularly, during a pandemic-related situation such as that caused by the SARS-CoV-2 virus, the need for these types of remote services becomes dramatically fundamental to reducing the movement of patients, and by extension reducing the risk of infection in healthcare settings. However, the recent innovation propogated by the cloud computing and internet of things (IoT) paradigms has been only partially taken into consideration by hospitals and, more generally, by medical centers so far. A crucial aspect that has slowed down wider adoption of these and other information and communication technology paradigms in hospitals is concern about the integrity, security, and privacy of exchanged data. Particularly within the healthcare domain, it is critical that shared clinical data be transmitted securely to prevent intentional or accidental illegal data manipulation. Furthermore, patients\u2019 privacy must be guaranteed.\nIn recent years, cloud computing and IoT paradigms, along with the concept of data federation, have been combined to form new technological approaches. One such approach is the concept of a federated cloud infrastructure, defined as a mesh of cloud providers that are interconnected to provide a universal decentralized computing environment where everything is driven by constraints and agreements in a ubiquitous, multi-provider infrastructure.[3] With the advent of IoT, an IoT-driven cloud paradim has also arisen. In this case, a set of smart embedded devices are interconnected with a remote cloud infrastructure, platform, or software into an internet-connected distributed system able to provide IoT as a service (IoTaaS). The natural evolution of this IoT-driven cloud concept was a federated ecosystem composed of small, medium, and large IoT-driven cloud providers able to collaborate in order to gain economies of scale and to enlarge their processing, storage, network, sensing, and actuating capabilities to arrange even more flexible IoTaaS.[4] The healthcare domain can benefit from these paradigms to improve clinical services and push down management costs through the creation of hospital IoT clouds[5] able to federate themselves.\nIn this paper, we focus on the medical laboratory as a case study. We examine the applied science laboratory, typically placed in a hospital or in a clinical center where clinical pathology exams are carried out on clinical samples to obtain information about the health of a patient to diagnose, treat, and prevent diseases. Blood tests (e.g., complete blood count [CBC], blood sugar, and so on) are performed by biomedical laboratory health technicians directly in the clinical laboratory, and results are validated and analyzed by doctors to guide diagnosis and treatment. More specifically, we examine these hospital labs under the emerging context of the IoT-connected laboratory operating in a federated cloud. In the case of these laboratories, clinical exams are performed on patients directly in a hospital by technicians who use IoT-enabled medical diagnostic devices integrated into the hospital's cloud system, with results automatically being sent via the hospital's cloud to doctors of federated hospitals for validation and\/or consultation. Biomedical laboratory health technicians, nurses, doctors and other clinical personnel belonging to different hospitals cooperate through their federated hospital clouds (FHCs) to form a virtual health team able to carry out a healthcare workflow.\nHowever, one of the major concerns about the accomplishment of such a workflow regards how to guarantee the non-repudiation and immutability of all health decisions.[6] In recent years, different solutions have been proposed to solve such an issue; among solutions is the option of using blockchain technology. Thanks to its intrinsic features of data non-repudiation and immutability, blockchain has aroused significant interest in both scientific and industrial communities. One of the major applications of blockchain is through the smart contract, i.e., a computer protocol aimed to digitally facilitate, verify, and enforce the negotiation of an agreement between subjects without the need of a certifying third party. Blockchain has been increasingly recognized as a technology able to address existing information access problems in different application domains, including healthcare. In fact, it can potentially enhance the perception of safety around medical operators, improving access to hospital cloud services that are guaranteed by greater transparency, security, privacy, traceability, and efficiency. In the scope of IoT-connected laboratories running in the cloud, smart contracts can make the transactions related to such a healthcare workflow traceable and irreversible.\nSpecifically, this paper proposes an architecture blueprint and a system prototype of an FHC service that addresses the healthcare workflow of an IoT-connected laboratory operating in the cloud. A special emphasis is given to blockchain, comparing both public and hybrid cloud network scenarios using the Ethereum platform to assess both processing time and economic cost. In particular, addressing the latter is vital as the Ethereum public network platform, available over the internet, requires that users (in our case, federated hospitals) pay a fee to perform each transaction.\nThe remainder of this paper is organized as follows. A brief overview of most recent initiatives about the adoption of blockchain in healthcare is provided in the next section. Motivations for pursuing an IoT-connected cloud approach are discussed in the subsequent section. A blueprint of FHC architecture is then presented, followed by a description of one of its possible implementations. Experiments specifically focusing on blockchain comparing public and hybrid cloud network scenarios are discussed in the penultimate section. The paper ends with conclusions and promising future developments.\n\nRelated work \nRecently, the role of blockchain technology in the healthcare domain has been surveyed in a number of scientific works.[7][8][9][10][11] More specifically, some works have demonstrated that blockchain can drastically improve the security of hospital information systems (HIS).[12][13][14][15] However, up to now, most of these scientific initiatives are either theoretical or at an early stage, and it is not always made clear which protocols and pieces of a framework should be used to carry out system implementations that can be deployed in real healthcare environments.\nBlockchain has been increasingly recognized as a tool able to address existing open information access issues.[16] In fact, it is possible to improve access to health services by using blockchain to achieve greater transparency, security, privacy, traceability, and efficiency. One such example can be found in the work of Ramani et al.[17], who describe a solution adopting blockchain with the purpose to guarantee authorized access to patients\u2019 medical information. In particular, mechanisms able to preserve both patient\u2019s identity and the integrity of their clinical history are proposed.[17] Another application of blockchain regards the supply chain in the pharmaceutical sector and the development of measures against counterfeit drugs. While the development of new drugs involves substantial costs related to studies to evaluate the safety and efficacy of the drug, the use of smart contracts guarantees informed consent procedures and allows for certifying the quality of the study data.[18] In another work, Shen et al.[19] propose an efficient data-sharing scheme called MedChain, which combines blockchain, digest chain, and structured P2P network techniques to overcome the efficiency issues in healthcare data sharing. Experiments with the system show that work can be performed with higher efficiency, while at the same time satisfying the security requirements of data sharing. Khatoon[20] examines different medical workflows which have been designed and implemented using the Ethereum blockchain platform, which involves complex medical procedures like surgery and clinical trials. In particular, Khatoon examines the smart contract system for healthcare management and estimates the associated costs in terms of feasibility. And finally, Satamraju[21] discusses a framework that integrates IoT networks with a blockchain to address potential privacy and security risks for data integrity in healthcare. In that work, a medical IoT device represented by a Raspberry Pi 3 Model B+ is attached to the patient\u2019s body to monitor their vitals, which are stored in an off-chain database that is accessed by the doctor, pharmacy, and insurance company via a DApp. All transactions take place utilizing smart contracts in a permissioned blockchain system implemented over Ethereum.\nDifferently from the aforementioned scientific initiatives\u2014which are mainly based on a public blockchain network approach\u2014this paper focuses on how a hybrid blockchain network approach (mixing both public and private) can be used to carry out the healthcare workflow of an IoT-connected laboratory running in a federated hospital cloud environment. Moreover, we demonstrate that our blockchain hybrid network approach reduces the number of required transactions while enhancing processing time and reducing economic cost.\n\nMotivation \nIn this section, after a brief overview of recent advances in diagnostic laboratory devices, we discuss the advantages of an IoT-connected cloud approach.\n\nRecent advancements in diagnostic laboratory devices \nA diagnostic laboratory device is defined here as medical equipment able to perform several blood tests, including a CBC, basic metabolic panel, complete metabolic panel, lipid panel, thyroid panel, enzyme markers, sexually transmitted infection tests, coagulation panel. and DHEA-sulfate serum test. Currently, there are many diagnostic laboratory devices available on the market, and they can be classified as \u201cconnected\u201d and \u201cnon-connected.\u201d Connected diagnostic laboratory devices have USB and network (wired and\/or wireless) interfaces and are able to export and send results to other devices. Non-connected diagnostic laboratory devices have no such interface for data transmission. \nWe provide a brief overview of the major connected diagnostic laboratory devices that are based on future IoT-connected laboratory services. The TeleMedCare Clinical Monitoring Unit (CMU)[22] is a medical device able to perform blood pressure, pulse oximetry, and blood glucose exams. Eversense[23] is a device able to perform continuous glucose monitoring via a chip that is installed subcutaneously on the patient and accessed with a mobile app. Med-Care[24] is an integrated solution for the automatic monitoring of glycemia that works with both web and mobile systems, and that can send alerts via email or SMS. HemoScreen[25] is a low-cost portable haematology analyzer which performs a complete blood count at the point of care, with results shown on a local web interface. The Samsung Labgeo PT10S[26] is a portable clinical chemistry analyzer that improves efficiency by saving time for clinicians and patients through fast, easy, and accurate blood analysis. It includes an ethernet interface for exporting exam result to an external personal computer (PC). It's worth noting that all the aforementioned devices require a blood sample in order to perform exams. However, alternative non-invasive experimental devices able to perform blood tests are the topic of ongoing study for both academic and industrial healthcare communities.\n\nTowards IoT-connected laboratories \nAn IoT-connected laboratory allows blood exams, results validation, diagnosis, and therapy assignment tasks to be performed in departments located in different hospitals. This is possible utilizing the creation of a virtual healthcare team composed of biomedical laboratory health technicians and doctors belonging to different federated hospitals. Cooperation is possible through the hospitals' FHCs. Hospital federation can involve several satellite clinics belonging either to the same healthcare organization or to different ones. An example of a healthcare organization that incorporates different hospitals is the provincial healthcare organization of Messina (Italy), also referred to as ASP Messina. As shown in Figure 1, it boasts eight health districts, including, Messina, Taromina, Milazzo, Lipari, Barcellona Pozzo di Gotto, Mistretta, and Sant\u2019Agata di Militello. \nThe health district of Lipari is located on the island of Lipari and provides a limited number of health services. It offers medical assistance to patients using an emergency room and a clinical pathology laboratory. Due to the limited number of health departments, patients with particular diseases are typically transferred to the nearby health districts of Milazzo or Barcellona Pozzo di Gotto (indeed by helicopter for urgent cases) if required. In this scenario, an IoT-connected laboratory service running on the cloud could improve clinical workflow by involving a virtual healthcare team, including technicians and doctors belonging, for example, to the Lipari, Milazzo, and Barcellona Pozzo di Gotto districts. In particular, blood tests could be performed by biomedical laboratory health technicians in the clinical pathology laboratory of Lipari, and results could be transmitted using the FHC ecosystem to a doctor of the Barcellona Pozzo di Gotto district for validation. Furthermore, leveraging the FHC environment, an additional consultation could be done with a doctor of the Milazzo district.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Federation of hospitals: clinical data is shared across participants for cooperation.\n\n\n\nUsing the term \u201chome hospital\u201d to define the hospital that is physically reached by the patient, the generic healthcare workflow accomplishing the aforementioned scenario implies the following phases:\n1. Hospitalization: Patient reaches a home hospital. Personal details, date, and type of visit are recorded. The patient is identified by a visit identification code. A doctor schedules all exams required to verify the nature of the disease. A virtual healthcare team is created involving technicians, nurses, and doctors belonging to the home hospital and other federated hospitals.\n2. Clinical Analysis: If required, a nurse of the home hospital takes a blood sample from the patient. A biomedical laboratory health technician of the home hospital performs blood tests, and results are saved by IoT medical devices automatically or by the technician manually on the FHC storage in a dedicated patient\u2019s directory.\n3. Validation: A doctor belonging to another federated hospital analyzes and validates the results of clinical analysis.\n4. Consultation: A selected pool of doctors belonging to the virtual healthcare team establish a teleconference to clarify the patient\u2019s clinical situation. The patient\u2019s health data and clinical analysis are shared among doctors, while still protecting sensitive data.\n5. Monitoring: The hospitalized patient is constantly monitored by nurses, who apply treatments based on therapeutic indications. Each treatment is recorded until the patient is released.\nNon-repudiation and immutability of all health decisions is a fundamental concern that must be addressed for the accomplishment of such a healthcare workflow. In this regard, the blockchain technology using smart contracts can make all transactions related to the healthcare workflow traceable and irreversible. In the remainder of this paper, we will focus on such an aspect.\n\nSystem design \nIt is important to guarantee that only authorized members of the virtual healthcare team are allowed to take actions; a wrong decision can lead to a worsening of clinical condition or the death of a patient. herefore, the FHC system has to guarantee that all actions performed by virtual healthcare team members are traceable and irreversible. To achieve such a goal, the following technologies are fundamental:\n\n Blockchain engine: To use the features of a decentralized and distributed certification system with the technology offered by the development and coding of a smart contract\n Cloud storage: To use an open-source and open-architecture file hosting service for file sharing managed with authorizations to archive all the files required to support the analytical testing of disease causes, including blood tests, computed tomography (CT) scans, and other laboratory tests\n NoSQL database: To exploit the potential of a document-oriented distributed database able to store and manage patient\u2019s data and diseases through tags for fast and efficient searching, and to store blockchain transaction hashes and links to files stored in cloud storage\nFigure 2 describes the FHC architecture. The system entry point of each hospital cloud is a dedicated Web Server Gateway Interface (WSGI) where pieces of electronic healthcare data are created manually by the clinical personnel or automatically collected by IoT devices that can be spread over different federated hospitals.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. IoT-connected federated hospital cloud (FHC) architecture\n\n\n\nA Data Anonymization Module (DAM) is responsible for hiding the patient\u2019s personal data in electronic health records (EHRs). This is possible by decoupling the patient\u2019s personal data from EHRs storing related pieces of information in different databases. To bind the EHR with the patient, a patient\u2019s anonymized identification number (patient_id) is generated from the DAM and stored within the EHR. Thus, only EHRs are shared between FHCs, whereas pieces of patients\u2019 personal data are never shared to preserve their privacy. All the produced clinical documentation, including EHR data, is uploaded to cloud storage with a patient_id to hide patient\u2019s data. A blockchain engine is responsible for storing information on the blockchain to certify data non-repudiation and immutability of treatment details guaranteeing accountability and authenticity. The transaction hash resulting from the mining process is stored in the NoSQL document-oriented database as an attribute of the treatment. A local database instance containing both patient and hospital personnel data is isolated from other federated hospitals because that information never needs to be shared. Treatments\u2019 details are anonymized and stored in a database shared with other participants in the FHC environment. The whole architecture is deployed using container virtualization to simplify installation, configuration, and maintenance in each FHC.\nHospitals belonging to a federation cooperate, ensuring that appropriate therapies and procedures are carried out. Figure 3 shows the sequence diagram describing an example of healthcare workflow, including three federated hospitals and where an IoT-connected laboratory service is provided. A generic patient who requires a medical visit reaches the emergency room of a home hospital that after evaluating the urgency of the case, identifies an available doctor. The patient is visited by the assigned doctor, who prescribes some sort of analytical testing such as blood tests, which can be done in the home hospital medical laboratory by a biomedical laboratory health technician. If the doctor responsible for the medical laboratory is not available (as with small hospital districts), a doctor belonging to the federated hospital (1) validates the results shared via federated cloud storage. Thence, such a doctor joins the virtual healthcare team, along with involved medical personnel of the home hospital. Should the doctor of the home hospital have a doubt about the therapy to prescribe, they can consult a doctor of a federated hospital (2) via teleconference. After this consultation, a therapy is assigned to the patient.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Sequence diagram describing an example of healthcare workflow accomplished in a FCH environment\n\n\n\nSystem prototype \nThe FHC architecture was designed to enable a virtual healthcare team to carry out every healthcare workflow, such as that described in the previous Section. Figure 4 shows the main software components of a possible system prototype implementation.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Hospital cloud software components\n\n\n\nAll requests coming from patients, nurses, technicians, and doctors flow through the WSGI interface developed with the Python web application framework Flask and deployed on the Gunicorn Python WSGI HTTP server. All the components are configured as Docker containers to take advantage of virtualization technology, allowing service portability, resiliency, and automatic updates that are typical of cloud-based infrastructure as a service (IaaS). The WSGI provides a front-end that allows retrieving all existing patients\u2019 information (such as personal details, disease and pharmaceutic codes, links to clinical documentation, and blockchain hash verification), adding new patients, and submiting new treatments specifying all the required pieces of information. Specifically, a web page is dedicated to the registration of a new patient for saving their primary personal information, and another web page is dedicated to the registration of a new treatment. It is possible to select the medical examination date, patient, and doctor who does the registration.\nAll the produced clinical documentation is uploaded in a local instance of NextCloud storage using a folder for each treatment, which does not contain any patient\u2019s personal data but rather a patient\u2019s anonymized identification number. Every change in the files or content of the folder is tracked, making it possible to keep a history of the documentation and its modifications. Recently, a few related works were published for data anonymization in a multicloud storage healthcare environment; however, these do not guarantee that pieces of data are traceable and irreversible.[27][28] Since patients\u2019 sensitive data must be anonymized and health records and treatments must be traceable and irreversible, related pieces of information are stored combining a MongoDB NoSQL database management system (DBMS) with the Ethereum blockchain platform. Therefore, all pieces of information are stored in both MongoDB and in the Ethereum network through smart contracts developed in Solidity. For experimental purposes (that will be discussed later), the Ethereum network was implemented in both public and hybrid configurations. In the first case, all FHCs share the public Ethereum network available over the internet, whereas, in the second case each FHC hosts a private Ethereum network to store local transactions and a public Ethereum network to synchronize the local transactions performed in each FHC.\nThe smart contract accepts the input parameters such as anonymized patient id and doctor id, disease, and pharmaceutic codes and stores these pieces of information in a simple data structure. The hash code resulting from the mining of each transaction is stored in the MongoDB database and can be used for verification using services like etherscan.io.\nThis service is capable of detecting any modification that occurred to files or a folder using a listener called External script. It is then possible to store the fingerprint and timestamp of each modification in the database, thus making it possible to track the history of each treatment. This feature is important to guarantee data integrity.\n\nExperiments \nCurrently, the most adopted blockchain configuration is based on a public network approach. As such, Ethereum is one of the major blockchain platforms. A public Ethereum instance is available over the internet and requires the payment of a fee for the execution of each transaction. However, the Ethereum platform can be also downloaded and installed in a private network. In this paper, using Ethereum, the objective of our experiments was to verify if the blockchain system of an FHC ecosystem including an IoT-connected laboratory service can be optimized in terms of both processing time and economic cost using the proposed hybrid network approach. Apart from processing time, it is important to highlight that considering the Ether (ETH) cryptocoin concurrency used in Ethereum, a small saving of ETH can result in putting aside a relevant amount of money (e.g., USD or EUR) in just a few months. In the following, we provide a description of both considered approaches.\n\n Ethereum public network: Each healthcare treatment is recorded in the public Ethereum blockchain network where time-to-mine and cost are subject to Ethereum network traffic (depending on the queue size, more time is required to extract transaction data, meaning higher costs for transaction management).\n Ethereum hybrid network: Each healthcare treatment is recorded in a private instance of an Ethereum blockchain network consisting of at least one node for each FHC and only one hash code, calculated as the MD5 of the last 100 concatenated treatments\u2019 transaction hash, and written in the public Ethereum blockchain network. In case the number of daily treatments is less than 100, the MD5 hash code is calculated as the concatenation of the last 24 hours' treatment transactions hash. The result of this is a negligible waiting queue and ETH cost but, on the other hand, there is a reduction of the mining power as a reduced number of miners are present in the private network as compared to the public one.\nSystem assessment was conducted analyzing the total execution time required to perform a varying number of transactions in healthcare workflows. Each FHC was simulated considering a server with following hardware\/software configuration: Intel Xeon E3-12xx v2 @ 2.7GHz, 4 core CPU, 4 GB RAM running Ubuntu Server 18.04. Each test was repeated 30 times considering 95% confidence intervals, and the average results were plotted. Table 1 summarizes experiment setup and average outcomes.\n\n\n\n\n\n\n\nTable 1. Summary of experiments performed\n\n\nParameter\n\nValue\n\n\nNumber of transactions tested\n\n[1; 100; 1000]\n\n\nTest executed for each run\n\n30\n\n\nConfidence interval\n\n95%\n\n\nGas price (Gwei)\n\n2\n\n\nAverage cost per transaction (ETH)\n\n0.0002\n\n\nAverage mining time (s)\n\n130\n\n\n\nFigure 5 shows the time-to-mine difference expressed in seconds between the two approaches, considering new treatment registration requests. On the x-axis we reported the number of treatment registration requests, whereas on the y-axis we reported the processing time expressed in seconds. Looking at the graph, we can observe that for roughly 80 treatment registration requests, both configurations present a similar trend, whereas by increasing the number of requests, the hybrid Ethereum network shows better performances than the public one thanks to the reduced waiting time in the mining queue.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Time comparison for public and hybrid Ethereum network approaches considering a varying number of treatment registration requests\n\n\n\nFigure 6 describes a cost comparison in Ether (ETH) for the two approaches. On the x-axis we reported the number of treatment registration requests, whereas on the y-axis we reported the cost expressed in Ether (ETH). From our tests, we appreciated that the average cost of a single transaction (i.e., a simple smart contract representing a new treatment memorization or modification) that has to be written in the public Ethereum blockchain is roughly 0.0002 ETH. It can be noted that for a small number of transactions there is not a perceptible convenience in preferring the proposed hybrid approach. This is because at least one transaction per day is written in the public Ethereum blockchain. However, it is clear that cost savings increases exponentially, increasing the number of transactions because only one public transaction out of every 100 private treatments will be paid with ETH cryptocurrency, resulting in an important cost-saving for the FHC ecosystem.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6. Cost comparison for public and hybrid Ethereum network approaches\n\n\n\nTest results demonstrate how the Ethereum hybrid network approach can be adopted to improve both processing time and cost savings, maintaining the same level of accountability and data certification as the public approach through certification on blockchain.\n\nConclusions and future work \nThis paper demonstrated how an IoT-connected laboratory service can be developed through a healthcare workflow running in an federated hospital cloud environment leveraging blockchain. Experimental results highlight that the performance of the Ethereum hybrid network certification system is improved in terms of cost and response time compared to an alternative public approach.\nWithout a doubt, blockchain technology is destined to evolve, improving system capabilities and robustness, and public test instances with different consensus protocols will be made available with benefits on performance and scalability.\nIn a pandemic-related situation such as that currently caused by the SARS-CoV-2 virus, the need for IoT-connected, cloud-based laboratory services becomes dramatically fundamental to reducing the movement of patients, and by extension reducing the risk of infection in healthcare settings. We hope that with this paper, we succeeded in stimulating the attention of both academic and industrial communities toward the adoption of Blockchain in the healthcare context to speed up the development of innovative services.\nIn the future, this work can be extended by integrating a comprehensive healthcare scenario with different involved organizations, such as a pharmaceutical company that registers in the blockchain all the phases of drug production, all the way to final packaging and shipment. In this case, when a patient buys a prescribed medicine, it would be possible to link the patient with the medicine box, which would mean an important step towards the end of falsified pharmaceuticals and an important assurance for the end user, who can be identified in case a specific drug package has been recalled.\n\nAcknowledgements \nAuthor contributions \nConceptualization, A.C., M.F., M.V., A.R. (Agata Romano); Methodology, A.R. (Armando Ruggeri), A.G.; Software; Validation, A.R. (Armando Ruggeri), A.G.; Formal Analysis A.R. (Armando Ruggeri); Investigation, A.C., M.F., M.V.; Resources, A.R. (Agata Romano), A.R. (Armando Ruggeri); Data Curation A.R. (Armando Ruggeri); Writing\u2014Original Draft Preparation A.C., A.R. (Agata Romano), A.R. (Armando Ruggeri); Writing\u2014Review & Editing, A.C., A.R. (Armando Ruggeri), M.F., M.V., A.R. (Agata Romano); Visualization, A.R. (Armando Ruggeri), A.G.; Supervision, A.C., M.V.; Project Administration, A.C., M.V.; Funding Acquisition, A.C., M.V. All authors have read and agreed to the published version of the manuscript.\n\nFunding \nThis research was funded by the Italian PON project \u201cTALISMAN,\u201d grant ARS01_01116 and by the Italian Healthcare Ministry Young Researcher project \u201cDo Severe acquired brain injury patients benefit from Telerehabilitation? A Cost-effectiveness analysis study,\u201d grant GR-2016-02361306.\n\nConflict of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 Hassenteufel, P.; Schweyer, F.-X.; Gerlinger, T. et al. (2020). \"The role of professional groups in policy change: Physician's organizations and the issue of local medical provision shortages in France and Germany\". European Policy Analysis 6 (1): 38\u201357. doi:10.1002\/epa2.1073.   \n\n\u2191 Dubas-Jak\u00f3bczyk, K.; Domaga\u0142a, A.; Mikos, M. (2019). \"Impact of the doctor deficit on hospital management in Poland: A mixed-method study\". International Journal of Health Planning and Management 34 (1): 187-195. doi:10.1002\/hpm.2612. PMID 30132977.   \n\n\u2191 Moreno-Vozmediano, R.; Huedo, E.; Llorente, I.M. et al. (2015). \"BEACON: A Cloud Network Federation Framework\". Proceedings of ESOCC 2015: Advances in Service-Oriented and Cloud Computing: 325\u201337. doi:10.1007\/978-3-319-33313-7_25.   \n\n\u2191 Celesti, A.; Fazio, M.; Giacobbe, M. et al. (2016). \"Characterizing Cloud Federation in IoT\". Proceedings of the 30th International Conference on Advanced Information Networking and Applications Workshops: 93\u201398. doi:10.1109\/WAINA.2016.152.   \n\n\u2191 Mulfari, D.; Celesti, A.; Puliafito, A. et al. (2013). \"How cloud computing can support on-demand assistive services\". Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility: 1\u20134. doi:10.1145\/2461121.2461140.   \n\n\u2191 Lian, J-W.; Yen, D.C.; Wang, Y.-T. (2014). \"An exploratory study to understand the critical factors affecting the decision to adopt cloud computing in Taiwan hospital\". International Journal of Information Management 34 (1): 28\u201336. doi:10.1016\/j.ijinfomgt.2013.09.004.   \n\n\u2191 Griggs, K.N.; Ossipova, O.; Kohlios, C.P. et al.. \"Healthcare Blockchain System Using Smart Contracts for Secure Automated Remote Patient Monitoring\". Journal of Medical Systems 42 (7): 130. doi:10.1007\/s10916-018-0982-x. PMID 29876661.   \n\n\u2191 Zubaydi, H.D.; Chong, Y.-W.; Ko, K. et al.. \"A Review on the Role of Blockchain Technology in the Healthcare Domain\". Electronics 8 (6): 679. doi:10.3390\/electronics8060679.   \n\n\u2191 Agbo, C.C.; Mahmoud. Q.H.; Eklund, J.M.. \"Blockchain Technology in Healthcare: A Systematic Review\". Healthcare 7 (2): 56. doi:10.3390\/healthcare7020056. PMC PMC6627742. PMID 30987333. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6627742 .   \n\n\u2191 H\u00f6lbl, M.; Kompara, M.; Kami\u0161ali\u0107, A. et al.. \"A Systematic Review of the Use of Blockchain in Healthcare\". Symmetry 10 (10): 470. doi:10.3390\/sym10100470.   \n\n\u2191 Qadri, Y.A.; Nauman, A.; Zikria, Y.B. et al.. \"The Future of Healthcare Internet of Things: A Survey of Emerging Technologies\". IEEE Communications Surveys & Tutorials 22 (2): 1121\u20131167. doi:10.1109\/COMST.2020.2973314.   \n\n\u2191 Chakraborty, S.; Aich, S.; Jim, H.-C.. \"A Secure Healthcare System Design Framework using Blockchain Technology\". Proceedings of the 21st International Conference on Advanced Communication Technology: 1260\u201364. doi:10.23919\/ICACT.2019.8701983.   \n\n\u2191 Dasaklis, T.K.; Casino, F.; Patsakis, C.. \"Blockchain Meets Smart Health: Towards Next Generation Healthcare Services\". Proceedings of the 9th International Conference on Information, Intelligence, Systems and Applications: 1\u20138. doi:10.1109\/IISA.2018.8633601.   \n\n\u2191 Srivastava, G.; Crichigno, J.; Dhar, S.. \"A Light and Secure Healthcare Blockchain for IoT Medical Devices\". Proceedings of the 2019 IEEE Canadian Conference of Electrical and Computer Engineering: 1\u20135. doi:10.1109\/CCECE.2019.8861593.   \n\n\u2191 Hossein, K.M.; Esmaeilli, M.E.; Dargahi, T. et al.. \"Blockchain-Based Privacy-Preserving Healthcare Architecture\". Proceedings of the 2019 IEEE Canadian Conference of Electrical and Computer Engineering: 1\u20134. doi:10.1109\/CCECE.2019.8861857.   \n\n\u2191 Zhang, P.; White, J.; Schmidt, D.C. et al.. \"FHIRChain: Applying Blockchain to Securely and Scalably Share Clinical Data\". Computational and Structural Biotechnology Journal 16: 267\u201378. doi:10.1016\/j.csbj.2018.07.004. PMC PMC6082774. PMID 30108685. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6082774 .   \n\n\u2191 17.0 17.1 Ramani, V.; Kumar, T.; Bracken, A. et al.. \"Secure and Efficient Data Accessibility in Blockchain Based Healthcare Systems\". Proceedings of the 2018 IEEE Global Communications Conference: 206\u201312. doi:10.1109\/GLOCOM.2018.8647221.   \n\n\u2191 Biggs, J.; Hinish, S.R.; Natale, M.A. et al. (2017). \"Blockchain: Revolutionizing the Global Supply Chain by Building Trust and Transparency\". Academia.edu. Rutgers University.   \n\n\u2191 Shen, B.; Guo, J.; Yang, Y.. \"MedChain: Efficient Healthcare Data Sharing via Blockchain\". Applied Sciences 9 (6): 1207. doi:10.3390\/app9061207.   \n\n\u2191 Khatoon, A.. \"A Blockchain-Based Smart Contract System for Healthcare Management\". Electronics 9 (1): 94. doi:10.3390\/electronics9010094.   \n\n\u2191 Satamraju, K.P.; Malarkodi, B.. \"Proof of Concept of Scalable Integration of Internet of Things and Blockchain in Healthcare\". Sensors 20 (5): 1389. doi:10.3390\/s20051389. PMC PMC7085612. PMID 32138380. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7085612 .   \n\n\u2191 \"TeleMedCare\". TeleMedCare. https:\/\/www.telemedcare.com\/ . Retrieved 20 March 2020 .   \n\n\u2191 \"Eversense\". Senseonics, Inc. https:\/\/www.eversensediabetes.com\/ . Retrieved 20 March 2020 .   \n\n\u2191 \"Med-Care\". T4A Srl. http:\/\/www.t4all.it\/portfolio-articoli\/med-care\/ . Retrieved 20 March 2020 .   \n\n\u2191 \"HemoScreen\". PixCell Medical Technologies Ltd. https:\/\/www.pixcell-medical.com\/hemoscreen\/ . Retrieved 20 March 2020 .   \n\n\u2191 \"LabGeo PT10\". Samsung. Archived from the original on 05 May 2020. https:\/\/web.archive.org\/web\/20200505231514\/https:\/\/samsunghealthcare.com\/en\/products\/InVitroDiagnostics\/Samsung%20LABGEO%20PT10\/Point-of-Care\/benefit .   \n\n\u2191 Galletta, A.; Bonanno, L.; Celesti, A. et al.. \"An approach to share MRI data over the Cloud preserving patients' privacy\". Proceedings of the 2017 IEEE Symposium on Computers and Communications: 94\u201399. doi:10.1109\/ISCC.2017.8024511.   \n\n\u2191 Galletta, A.; Celesti, A.; Tusa, F. et al.. \"Big MRI Data Dissemination and Retrieval in a Multi-Cloud Hospital Storage System\". Proceedings of the 2017 International Conference on Digital Health: 162\u201366. doi:10.1145\/3079452.3079507.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. The original title\u2014Blockchain-Based Healthcare Workflow for Tele-Medical Laboratory in Federated Hospital IoT Clouds\u2014was changed slightly as \"tele-medical laboratory\" is a confusing phrase that elicits thoughts of \"telemedicine,\" which is not an accurate portrayal of the article. The new title intends to capture the true spirit of the concept, and this change was also propogated throughout this version of the article. In some cases important information was missing from the references, and that information was added. The Samsung Labgeo PT10S has since been discontinued; an archived version of the URL for it is used here.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\">https:\/\/www.limswiki.org\/index.php\/Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2020)LIMSwiki journal articles (all)LIMSwiki journal articles on cloud computingLIMSwiki journal articles on health informaticsLIMSwiki journal articles on internet of things\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 11 May 2021, at 17:45.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 488 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","e49209f2666c8e463be219dc620f0ec5_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Blockchain-based healthcare workflow for IoT-connected laboratories in federated hospital clouds<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>In a <a href=\"https:\/\/www.limswiki.org\/index.php\/Pandemic\" title=\"Pandemic\" class=\"wiki-link\" data-key=\"bd9a48e6c6e41b6d603ee703836b01f1\">pandemic<\/a>-related situation such as that caused by the <a href=\"https:\/\/www.limswiki.org\/index.php\/SARS-CoV-2\" title=\"SARS-CoV-2\" class=\"mw-redirect wiki-link\" data-key=\"6c6b78479de2c640028696ed32948526\">SARS-CoV-2<\/a> virus, the need for <a href=\"https:\/\/www.limswiki.org\/index.php\/Telemedicine\" title=\"Telemedicine\" class=\"wiki-link\" data-key=\"d2cc9ab69dbfb679bcee20472b08fe93\">telemedicine<\/a> and other distanced services becomes dramatically fundamental to reducing the movement of patients, and by extension reducing the risk of infection in healthcare settings. One potential avenue for achieving this is through leveraging <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud computing<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Internet_of_things\" title=\"Internet of things\" class=\"wiki-link\" data-key=\"13e0b826fa1770fe4bea72e3cb942f0f\">internet of things<\/a> (IoT) technologies. This paper proposes an IoT-connected laboratory service where clinical exams are performed on patients directly in a <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> by technicians through the use of IoT medical diagnostic devices, with results automatically being sent via the hospital's cloud to doctors of federated hospitals for validation and\/or consultation. In particular, we discuss a distributed scenario where nurses, technicians, and medical doctors belonging to different hospitals cooperate through their federated hospital clouds to form a virtual health team able to carry out a healthcare <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a> in secure fashion by leveraging the intrinsic security features of <a href=\"https:\/\/www.limswiki.org\/index.php\/Blockchain\" title=\"Blockchain\" class=\"wiki-link\" data-key=\"ae8b186c311716aca561aaee91944f8e\">blockchain<\/a> technology. In particular, both public and hybrid blockchain scenarios are discussed and assessed using the Ethereum platform.\n<\/p><p><b>Keywords<\/b>: blockchain, smart contract, workflow, healthcare, hospital, cloud computing, IoT, data federation\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Recent advancements in information and communication technology have paved the way toward new innovative <a href=\"https:\/\/www.limswiki.org\/index.php\/Telemedicine\" title=\"Telemedicine\" class=\"wiki-link\" data-key=\"d2cc9ab69dbfb679bcee20472b08fe93\">telemedicine<\/a> and other remote healthcare services, which are able to support the growing demand of even more accessible medical treatments.<sup id=\"rdp-ebb-cite_ref-HassenteufelTheRole20_1-0\" class=\"reference\"><a href=\"#cite_note-HassenteufelTheRole20-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Dubas-Jak.C3.B3bczykImpact19_2-0\" class=\"reference\"><a href=\"#cite_note-Dubas-Jak.C3.B3bczykImpact19-2\">[2]<\/a><\/sup> Particularly, during a <a href=\"https:\/\/www.limswiki.org\/index.php\/Pandemic\" title=\"Pandemic\" class=\"wiki-link\" data-key=\"bd9a48e6c6e41b6d603ee703836b01f1\">pandemic<\/a>-related situation such as that caused by the <a href=\"https:\/\/www.limswiki.org\/index.php\/SARS-CoV-2\" title=\"SARS-CoV-2\" class=\"mw-redirect wiki-link\" data-key=\"6c6b78479de2c640028696ed32948526\">SARS-CoV-2<\/a> virus, the need for these types of remote services becomes dramatically fundamental to reducing the movement of patients, and by extension reducing the risk of infection in healthcare settings. However, the recent innovation propogated by the <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud computing<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Internet_of_things\" title=\"Internet of things\" class=\"wiki-link\" data-key=\"13e0b826fa1770fe4bea72e3cb942f0f\">internet of things<\/a> (IoT) paradigms has been only partially taken into consideration by <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospitals<\/a> and, more generally, by medical centers so far. A crucial aspect that has slowed down wider adoption of these and other information and communication technology paradigms in hospitals is concern about the <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">integrity<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_security\" title=\"Information security\" class=\"wiki-link\" data-key=\"9eff362d944224ff1d4ffe3a149d7cff\">security<\/a>, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_privacy\" title=\"Information privacy\" class=\"wiki-link\" data-key=\"185f6d9f874e48914b5789317408f782\">privacy<\/a> of exchanged data. Particularly within the healthcare domain, it is critical that shared clinical data be transmitted securely to prevent intentional or accidental illegal data manipulation. Furthermore, patients\u2019 privacy must be guaranteed.\n<\/p><p>In recent years, cloud computing and IoT paradigms, along with the concept of data federation, have been combined to form new technological approaches. One such approach is the concept of a federated cloud infrastructure, defined as a mesh of cloud providers that are interconnected to provide a universal decentralized computing environment where everything is driven by constraints and agreements in a ubiquitous, multi-provider infrastructure.<sup id=\"rdp-ebb-cite_ref-Moreno-VozmedianoBEACON16_3-0\" class=\"reference\"><a href=\"#cite_note-Moreno-VozmedianoBEACON16-3\">[3]<\/a><\/sup> With the advent of IoT, an IoT-driven cloud paradim has also arisen. In this case, a set of smart embedded devices are interconnected with a remote cloud infrastructure, platform, or software into an internet-connected distributed system able to provide IoT as a service (IoTaaS). The natural evolution of this IoT-driven cloud concept was a federated ecosystem composed of small, medium, and large IoT-driven cloud providers able to collaborate in order to gain economies of scale and to enlarge their processing, storage, network, sensing, and actuating capabilities to arrange even more flexible IoTaaS.<sup id=\"rdp-ebb-cite_ref-CelestiCharact16_4-0\" class=\"reference\"><a href=\"#cite_note-CelestiCharact16-4\">[4]<\/a><\/sup> The healthcare domain can benefit from these paradigms to improve clinical services and push down management costs through the creation of hospital IoT clouds<sup id=\"rdp-ebb-cite_ref-MulfariHowCloud13_5-0\" class=\"reference\"><a href=\"#cite_note-MulfariHowCloud13-5\">[5]<\/a><\/sup> able to federate themselves.\n<\/p><p>In this paper, we focus on the <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_laboratory\" title=\"Clinical laboratory\" class=\"wiki-link\" data-key=\"307bcdf1bdbcd1bb167cee435b7a5463\">medical laboratory<\/a> as a case study. We examine the applied science laboratory, typically placed in a hospital or in a clinical center where <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_pathology\" title=\"Clinical pathology\" class=\"wiki-link\" data-key=\"9b0b91aa13ae0209e073db6f30493daf\">clinical pathology<\/a> exams are carried out on clinical <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">samples<\/a> to obtain <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> about the health of a patient to diagnose, treat, and prevent diseases. Blood tests (e.g., complete blood count [CBC], blood sugar, and so on) are performed by biomedical laboratory health technicians directly in the clinical laboratory, and results are validated and analyzed by doctors to guide diagnosis and treatment. More specifically, we examine these hospital labs under the emerging context of the IoT-connected laboratory operating in a federated cloud. In the case of these laboratories, clinical exams are performed on patients directly in a hospital by technicians who use IoT-enabled medical diagnostic devices integrated into the hospital's cloud system, with results automatically being sent via the hospital's cloud to doctors of federated hospitals for validation and\/or consultation. Biomedical laboratory health technicians, nurses, doctors and other clinical personnel belonging to different hospitals cooperate through their federated hospital clouds (FHCs) to form a virtual health team able to carry out a healthcare <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a>.\n<\/p><p>However, one of the major concerns about the accomplishment of such a workflow regards how to guarantee the non-repudiation and immutability of all health decisions.<sup id=\"rdp-ebb-cite_ref-LianAnExplor14_6-0\" class=\"reference\"><a href=\"#cite_note-LianAnExplor14-6\">[6]<\/a><\/sup> In recent years, different solutions have been proposed to solve such an issue; among solutions is the option of using <a href=\"https:\/\/www.limswiki.org\/index.php\/Blockchain\" title=\"Blockchain\" class=\"wiki-link\" data-key=\"ae8b186c311716aca561aaee91944f8e\">blockchain<\/a> technology. Thanks to its intrinsic features of data non-repudiation and immutability, blockchain has aroused significant interest in both scientific and industrial communities. One of the major applications of blockchain is through the smart contract, i.e., a computer protocol aimed to digitally facilitate, verify, and enforce the negotiation of an agreement between subjects without the need of a certifying third party. Blockchain has been increasingly recognized as a technology able to address existing information access problems in different application domains, including healthcare. In fact, it can potentially enhance the perception of safety around medical operators, improving access to hospital cloud services that are guaranteed by greater transparency, security, privacy, traceability, and efficiency. In the scope of IoT-connected laboratories running in the cloud, smart contracts can make the transactions related to such a healthcare workflow traceable and irreversible.\n<\/p><p>Specifically, this paper proposes an architecture blueprint and a system prototype of an FHC service that addresses the healthcare workflow of an IoT-connected laboratory operating in the cloud. A special emphasis is given to blockchain, comparing both public and hybrid cloud network scenarios using the Ethereum platform to assess both processing time and economic cost. In particular, addressing the latter is vital as the Ethereum public network platform, available over the internet, requires that users (in our case, federated hospitals) pay a fee to perform each transaction.\n<\/p><p>The remainder of this paper is organized as follows. A brief overview of most recent initiatives about the adoption of blockchain in healthcare is provided in the next section. Motivations for pursuing an IoT-connected cloud approach are discussed in the subsequent section. A blueprint of FHC architecture is then presented, followed by a description of one of its possible implementations. Experiments specifically focusing on blockchain comparing public and hybrid cloud network scenarios are discussed in the penultimate section. The paper ends with conclusions and promising future developments.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Related_work\">Related work<\/span><\/h2>\n<p>Recently, the role of blockchain technology in the healthcare domain has been surveyed in a number of scientific works.<sup id=\"rdp-ebb-cite_ref-GriggsHealth18_7-0\" class=\"reference\"><a href=\"#cite_note-GriggsHealth18-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZubaydiARev19_8-0\" class=\"reference\"><a href=\"#cite_note-ZubaydiARev19-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AgboBlock19_9-0\" class=\"reference\"><a href=\"#cite_note-AgboBlock19-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-H.C3.B6lblASys18_10-0\" class=\"reference\"><a href=\"#cite_note-H.C3.B6lblASys18-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-QadriTheFut20_11-0\" class=\"reference\"><a href=\"#cite_note-QadriTheFut20-11\">[11]<\/a><\/sup> More specifically, some works have demonstrated that blockchain can drastically improve the security of <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital_information_system\" title=\"Hospital information system\" class=\"wiki-link\" data-key=\"d8385de7b1f39a39d793f8ce349b448d\">hospital information systems<\/a> (HIS).<sup id=\"rdp-ebb-cite_ref-ChakrabortyASec19_12-0\" class=\"reference\"><a href=\"#cite_note-ChakrabortyASec19-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DasaklisBlock18_13-0\" class=\"reference\"><a href=\"#cite_note-DasaklisBlock18-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SrivastavaALight19_14-0\" class=\"reference\"><a href=\"#cite_note-SrivastavaALight19-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HosseinBlock19_15-0\" class=\"reference\"><a href=\"#cite_note-HosseinBlock19-15\">[15]<\/a><\/sup> However, up to now, most of these scientific initiatives are either theoretical or at an early stage, and it is not always made clear which protocols and pieces of a framework should be used to carry out system implementations that can be deployed in real healthcare environments.\n<\/p><p>Blockchain has been increasingly recognized as a tool able to address existing open information access issues.<sup id=\"rdp-ebb-cite_ref-ZhangFHIR18_16-0\" class=\"reference\"><a href=\"#cite_note-ZhangFHIR18-16\">[16]<\/a><\/sup> In fact, it is possible to improve access to health services by using blockchain to achieve greater transparency, security, privacy, traceability, and efficiency. One such example can be found in the work of Ramani <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-RamaniSecure18_17-0\" class=\"reference\"><a href=\"#cite_note-RamaniSecure18-17\">[17]<\/a><\/sup>, who describe a solution adopting blockchain with the purpose to guarantee authorized access to patients\u2019 medical information. In particular, mechanisms able to preserve both patient\u2019s identity and the integrity of their clinical history are proposed.<sup id=\"rdp-ebb-cite_ref-RamaniSecure18_17-1\" class=\"reference\"><a href=\"#cite_note-RamaniSecure18-17\">[17]<\/a><\/sup> Another application of blockchain regards the supply chain in the pharmaceutical sector and the development of measures against counterfeit drugs. While the development of new drugs involves substantial costs related to studies to evaluate the safety and efficacy of the drug, the use of smart contracts guarantees informed consent procedures and allows for certifying the quality of the study data.<sup id=\"rdp-ebb-cite_ref-BiggsBlock17_18-0\" class=\"reference\"><a href=\"#cite_note-BiggsBlock17-18\">[18]<\/a><\/sup> In another work, Shen <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ShenMedChain19_19-0\" class=\"reference\"><a href=\"#cite_note-ShenMedChain19-19\">[19]<\/a><\/sup> propose an efficient data-sharing scheme called MedChain, which combines blockchain, digest chain, and structured P2P network techniques to overcome the efficiency issues in healthcare data sharing. Experiments with the system show that work can be performed with higher efficiency, while at the same time satisfying the security requirements of data sharing. Khatoon<sup id=\"rdp-ebb-cite_ref-KhatoonABlock20_20-0\" class=\"reference\"><a href=\"#cite_note-KhatoonABlock20-20\">[20]<\/a><\/sup> examines different medical workflows which have been designed and implemented using the Ethereum blockchain platform, which involves complex medical procedures like surgery and clinical trials. In particular, Khatoon examines the smart contract system for healthcare management and estimates the associated costs in terms of feasibility. And finally, Satamraju<sup id=\"rdp-ebb-cite_ref-SatamrajuProof20_21-0\" class=\"reference\"><a href=\"#cite_note-SatamrajuProof20-21\">[21]<\/a><\/sup> discusses a framework that integrates IoT networks with a blockchain to address potential privacy and security risks for data integrity in healthcare. In that work, a medical IoT device represented by a Raspberry Pi 3 Model B+ is attached to the patient\u2019s body to monitor their vitals, which are stored in an off-chain database that is accessed by the doctor, pharmacy, and insurance company via a DApp. All transactions take place utilizing smart contracts in a permissioned blockchain system implemented over Ethereum.\n<\/p><p>Differently from the aforementioned scientific initiatives\u2014which are mainly based on a public blockchain network approach\u2014this paper focuses on how a hybrid blockchain network approach (mixing both public and private) can be used to carry out the healthcare workflow of an IoT-connected laboratory running in a federated hospital cloud environment. Moreover, we demonstrate that our blockchain hybrid network approach reduces the number of required transactions while enhancing processing time and reducing economic cost.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Motivation\">Motivation<\/span><\/h2>\n<p>In this section, after a brief overview of recent advances in diagnostic laboratory devices, we discuss the advantages of an IoT-connected cloud approach.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Recent_advancements_in_diagnostic_laboratory_devices\">Recent advancements in diagnostic laboratory devices<\/span><\/h3>\n<p>A diagnostic laboratory device is defined here as medical equipment able to perform several blood tests, including a CBC, basic metabolic panel, complete metabolic panel, lipid panel, thyroid panel, enzyme markers, sexually transmitted infection tests, coagulation panel. and DHEA-sulfate serum test. Currently, there are many diagnostic laboratory devices available on the market, and they can be classified as \u201cconnected\u201d and \u201cnon-connected.\u201d Connected diagnostic laboratory devices have USB and network (wired and\/or wireless) interfaces and are able to export and send results to other devices. Non-connected diagnostic laboratory devices have no such interface for data transmission. \n<\/p><p>We provide a brief overview of the major connected diagnostic laboratory devices that are based on future IoT-connected laboratory services. The TeleMedCare Clinical Monitoring Unit (CMU)<sup id=\"rdp-ebb-cite_ref-TMCHome_22-0\" class=\"reference\"><a href=\"#cite_note-TMCHome-22\">[22]<\/a><\/sup> is a medical device able to perform blood pressure, pulse oximetry, and blood glucose exams. Eversense<sup id=\"rdp-ebb-cite_ref-EverHome_23-0\" class=\"reference\"><a href=\"#cite_note-EverHome-23\">[23]<\/a><\/sup> is a device able to perform continuous glucose monitoring via a chip that is installed subcutaneously on the patient and accessed with a mobile app. Med-Care<sup id=\"rdp-ebb-cite_ref-Med-Care_24-0\" class=\"reference\"><a href=\"#cite_note-Med-Care-24\">[24]<\/a><\/sup> is an integrated solution for the automatic monitoring of glycemia that works with both web and mobile systems, and that can send alerts via email or SMS. HemoScreen<sup id=\"rdp-ebb-cite_ref-HemoScreen_25-0\" class=\"reference\"><a href=\"#cite_note-HemoScreen-25\">[25]<\/a><\/sup> is a low-cost portable haematology analyzer which performs a complete blood count at the point of care, with results shown on a local web interface. The Samsung Labgeo PT10S<sup id=\"rdp-ebb-cite_ref-LabGeoPT10_26-0\" class=\"reference\"><a href=\"#cite_note-LabGeoPT10-26\">[26]<\/a><\/sup> is a portable clinical chemistry analyzer that improves efficiency by saving time for clinicians and patients through fast, easy, and accurate blood analysis. It includes an ethernet interface for exporting exam result to an external personal computer (PC). It's worth noting that all the aforementioned devices require a blood sample in order to perform exams. However, alternative non-invasive experimental devices able to perform blood tests are the topic of ongoing study for both academic and industrial healthcare communities.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Towards_IoT-connected_laboratories\">Towards IoT-connected laboratories<\/span><\/h3>\n<p>An IoT-connected laboratory allows blood exams, results validation, diagnosis, and therapy assignment tasks to be performed in departments located in different hospitals. This is possible utilizing the creation of a virtual healthcare team composed of biomedical laboratory health technicians and doctors belonging to different federated hospitals. Cooperation is possible through the hospitals' FHCs. Hospital federation can involve several satellite clinics belonging either to the same healthcare organization or to different ones. An example of a healthcare organization that incorporates different hospitals is the provincial healthcare organization of Messina (Italy), also referred to as ASP Messina. As shown in Figure 1, it boasts eight health districts, including, Messina, Taromina, Milazzo, Lipari, Barcellona Pozzo di Gotto, Mistretta, and Sant\u2019Agata di Militello. \n<\/p><p>The health district of Lipari is located on the island of Lipari and provides a limited number of health services. It offers medical assistance to patients using an emergency room and a clinical pathology laboratory. Due to the limited number of health departments, patients with particular diseases are typically transferred to the nearby health districts of Milazzo or Barcellona Pozzo di Gotto (indeed by helicopter for urgent cases) if required. In this scenario, an IoT-connected laboratory service running on the cloud could improve clinical workflow by involving a virtual healthcare team, including technicians and doctors belonging, for example, to the Lipari, Milazzo, and Barcellona Pozzo di Gotto districts. In particular, blood tests could be performed by biomedical laboratory health technicians in the clinical pathology laboratory of Lipari, and results could be transmitted using the FHC ecosystem to a doctor of the Barcellona Pozzo di Gotto district for validation. Furthermore, leveraging the FHC environment, an additional consultation could be done with a doctor of the Milazzo district.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Celesti_Sensors20_20-9.png\" class=\"image wiki-link\" data-key=\"7bf18c5fc36372c325a3f53df7032ebf\"><img alt=\"Fig1 Celesti Sensors20 20-9.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/e6\/Fig1_Celesti_Sensors20_20-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Federation of hospitals: clinical data is shared across participants for cooperation.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Using the term \u201chome hospital\u201d to define the hospital that is physically reached by the patient, the generic healthcare workflow accomplishing the aforementioned scenario implies the following phases:\n<\/p><p>1. Hospitalization: Patient reaches a home hospital. Personal details, date, and type of visit are recorded. The patient is identified by a visit identification code. A doctor schedules all exams required to verify the nature of the disease. A virtual healthcare team is created involving technicians, nurses, and doctors belonging to the home hospital and other federated hospitals.\n<\/p><p>2. Clinical Analysis: If required, a nurse of the home hospital takes a blood sample from the patient. A biomedical laboratory health technician of the home hospital performs blood tests, and results are saved by IoT medical devices automatically or by the technician manually on the FHC storage in a dedicated patient\u2019s directory.\n<\/p><p>3. Validation: A doctor belonging to another federated hospital analyzes and validates the results of clinical analysis.\n<\/p><p>4. Consultation: A selected pool of doctors belonging to the virtual healthcare team establish a teleconference to clarify the patient\u2019s clinical situation. The patient\u2019s health data and clinical analysis are shared among doctors, while still protecting sensitive data.\n<\/p><p>5. Monitoring: The hospitalized patient is constantly monitored by nurses, who apply treatments based on therapeutic indications. Each treatment is recorded until the patient is released.\n<\/p><p>Non-repudiation and immutability of all health decisions is a fundamental concern that must be addressed for the accomplishment of such a healthcare workflow. In this regard, the blockchain technology using smart contracts can make all transactions related to the healthcare workflow traceable and irreversible. In the remainder of this paper, we will focus on such an aspect.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"System_design\">System design<\/span><\/h2>\n<p>It is important to guarantee that only authorized members of the virtual healthcare team are allowed to take actions; a wrong decision can lead to a worsening of clinical condition or the death of a patient. herefore, the FHC system has to guarantee that all actions performed by virtual healthcare team members are traceable and irreversible. To achieve such a goal, the following technologies are fundamental:\n<\/p>\n<ul><li> Blockchain engine: To use the features of a decentralized and distributed certification system with the technology offered by the development and coding of a smart contract<\/li>\n<li> Cloud storage: To use an open-source and open-architecture file hosting service for file sharing managed with authorizations to archive all the files required to support the analytical testing of disease causes, including blood tests, computed tomography (CT) scans, and other laboratory tests<\/li>\n<li> NoSQL database: To exploit the potential of a document-oriented distributed database able to store and manage patient\u2019s data and diseases through tags for fast and efficient searching, and to store blockchain transaction hashes and links to files stored in cloud storage<\/li><\/ul>\n<p>Figure 2 describes the FHC architecture. The system entry point of each hospital cloud is a dedicated Web Server Gateway Interface (WSGI) where pieces of electronic healthcare data are created manually by the clinical personnel or automatically collected by IoT devices that can be spread over different federated hospitals.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Celesti_Sensors20_20-9.png\" class=\"image wiki-link\" data-key=\"6da29bbe0c19156706abf773fd63d379\"><img alt=\"Fig2 Celesti Sensors20 20-9.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/57\/Fig2_Celesti_Sensors20_20-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> IoT-connected federated hospital cloud (FHC) architecture<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>A Data Anonymization Module (DAM) is responsible for hiding the patient\u2019s personal data in (EHRs). This is possible by decoupling the patient\u2019s personal data from EHRs storing related pieces of information in different databases. To bind the EHR with the patient, a patient\u2019s anonymized identification number (patient_id) is generated from the DAM and stored within the EHR. Thus, only EHRs are shared between FHCs, whereas pieces of patients\u2019 personal data are never shared to preserve their privacy. All the produced clinical documentation, including EHR data, is uploaded to cloud storage with a patient_id to hide patient\u2019s data. A blockchain engine is responsible for storing information on the blockchain to certify data non-repudiation and immutability of treatment details guaranteeing accountability and authenticity. The transaction hash resulting from the mining process is stored in the NoSQL document-oriented database as an attribute of the treatment. A local database instance containing both patient and hospital personnel data is isolated from other federated hospitals because that information never needs to be shared. Treatments\u2019 details are anonymized and stored in a database shared with other participants in the FHC environment. The whole architecture is deployed using container virtualization to simplify installation, configuration, and maintenance in each FHC.\n<\/p><p>Hospitals belonging to a federation cooperate, ensuring that appropriate therapies and procedures are carried out. Figure 3 shows the sequence diagram describing an example of healthcare workflow, including three federated hospitals and where an IoT-connected laboratory service is provided. A generic patient who requires a medical visit reaches the emergency room of a home hospital that after evaluating the urgency of the case, identifies an available doctor. The patient is visited by the assigned doctor, who prescribes some sort of analytical testing such as blood tests, which can be done in the home hospital medical laboratory by a biomedical laboratory health technician. If the doctor responsible for the medical laboratory is not available (as with small hospital districts), a doctor belonging to the federated hospital (1) validates the results shared via federated cloud storage. Thence, such a doctor joins the virtual healthcare team, along with involved medical personnel of the home hospital. Should the doctor of the home hospital have a doubt about the therapy to prescribe, they can consult a doctor of a federated hospital (2) via teleconference. After this consultation, a therapy is assigned to the patient.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Celesti_Sensors20_20-9.png\" class=\"image wiki-link\" data-key=\"ad808547399ff408a05eed1d65566b2f\"><img alt=\"Fig3 Celesti Sensors20 20-9.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/c8\/Fig3_Celesti_Sensors20_20-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Sequence diagram describing an example of healthcare workflow accomplished in a FCH environment<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"System_prototype\">System prototype<\/span><\/h2>\n<p>The FHC architecture was designed to enable a virtual healthcare team to carry out every healthcare workflow, such as that described in the previous Section. Figure 4 shows the main software components of a possible system prototype implementation.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Celesti_Sensors20_20-9.png\" class=\"image wiki-link\" data-key=\"6ca7243afaa2f187c8d97b972c6c27d7\"><img alt=\"Fig4 Celesti Sensors20 20-9.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f3\/Fig4_Celesti_Sensors20_20-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Hospital cloud software components<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>All requests coming from patients, nurses, technicians, and doctors flow through the WSGI interface developed with the Python web application framework Flask and deployed on the Gunicorn Python WSGI HTTP server. All the components are configured as Docker containers to take advantage of virtualization technology, allowing service portability, resiliency, and automatic updates that are typical of cloud-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Infrastructure_as_a_service\" title=\"Infrastructure as a service\" class=\"wiki-link\" data-key=\"70b93c66f23363688d45f0d354e5c032\">infrastructure as a service<\/a> (IaaS). The WSGI provides a front-end that allows retrieving all existing patients\u2019 information (such as personal details, disease and pharmaceutic codes, links to clinical documentation, and blockchain hash verification), adding new patients, and submiting new treatments specifying all the required pieces of information. Specifically, a web page is dedicated to the registration of a new patient for saving their primary personal information, and another web page is dedicated to the registration of a new treatment. It is possible to select the medical examination date, patient, and doctor who does the registration.\n<\/p><p>All the produced clinical documentation is uploaded in a local instance of NextCloud storage using a folder for each treatment, which does not contain any patient\u2019s personal data but rather a patient\u2019s anonymized identification number. Every change in the files or content of the folder is tracked, making it possible to keep a history of the documentation and its modifications. Recently, a few related works were published for data anonymization in a multicloud storage healthcare environment; however, these do not guarantee that pieces of data are traceable and irreversible.<sup id=\"rdp-ebb-cite_ref-GallettaAnApp17_27-0\" class=\"reference\"><a href=\"#cite_note-GallettaAnApp17-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GallettaBigMRI17_28-0\" class=\"reference\"><a href=\"#cite_note-GallettaBigMRI17-28\">[28]<\/a><\/sup> Since patients\u2019 sensitive data must be anonymized and health records and treatments must be traceable and irreversible, related pieces of information are stored combining a MongoDB NoSQL database management system (DBMS) with the Ethereum blockchain platform. Therefore, all pieces of information are stored in both MongoDB and in the Ethereum network through smart contracts developed in Solidity. For experimental purposes (that will be discussed later), the Ethereum network was implemented in both public and hybrid configurations. In the first case, all FHCs share the public Ethereum network available over the internet, whereas, in the second case each FHC hosts a private Ethereum network to store local transactions and a public Ethereum network to synchronize the local transactions performed in each FHC.\n<\/p><p>The smart contract accepts the input parameters such as anonymized patient id and doctor id, disease, and pharmaceutic codes and stores these pieces of information in a simple data structure. The hash code resulting from the mining of each transaction is stored in the MongoDB database and can be used for verification using services like etherscan.io.\n<\/p><p>This service is capable of detecting any modification that occurred to files or a folder using a listener called External script. It is then possible to store the fingerprint and timestamp of each modification in the database, thus making it possible to track the history of each treatment. This feature is important to guarantee data integrity.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Experiments\">Experiments<\/span><\/h2>\n<p>Currently, the most adopted blockchain configuration is based on a public network approach. As such, Ethereum is one of the major blockchain platforms. A public Ethereum instance is available over the internet and requires the payment of a fee for the execution of each transaction. However, the Ethereum platform can be also downloaded and installed in a private network. In this paper, using Ethereum, the objective of our experiments was to verify if the blockchain system of an FHC ecosystem including an IoT-connected laboratory service can be optimized in terms of both processing time and economic cost using the proposed hybrid network approach. Apart from processing time, it is important to highlight that considering the Ether (ETH) cryptocoin concurrency used in Ethereum, a small saving of ETH can result in putting aside a relevant amount of money (e.g., USD or EUR) in just a few months. In the following, we provide a description of both considered approaches.\n<\/p>\n<ul><li> Ethereum public network: Each healthcare treatment is recorded in the public Ethereum blockchain network where time-to-mine and cost are subject to Ethereum network traffic (depending on the queue size, more time is required to extract transaction data, meaning higher costs for transaction management).<\/li>\n<li> Ethereum hybrid network: Each healthcare treatment is recorded in a private instance of an Ethereum blockchain network consisting of at least one node for each FHC and only one hash code, calculated as the MD5 of the last 100 concatenated treatments\u2019 transaction hash, and written in the public Ethereum blockchain network. In case the number of daily treatments is less than 100, the MD5 hash code is calculated as the concatenation of the last 24 hours' treatment transactions hash. The result of this is a negligible waiting queue and ETH cost but, on the other hand, there is a reduction of the mining power as a reduced number of miners are present in the private network as compared to the public one.<\/li><\/ul>\n<p>System assessment was conducted analyzing the total execution time required to perform a varying number of transactions in healthcare workflows. Each FHC was simulated considering a server with following hardware\/software configuration: Intel Xeon E3-12xx v2 @ 2.7GHz, 4 core CPU, 4 GB RAM running Ubuntu Server 18.04. Each test was repeated 30 times considering 95% confidence intervals, and the average results were plotted. Table 1 summarizes experiment setup and average outcomes.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b> Summary of experiments performed\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Parameter\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Value\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Number of transactions tested\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">[1; 100; 1000]\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Test executed for each run\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Confidence interval\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">95%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Gas price (Gwei)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Average cost per transaction (ETH)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.0002\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Average mining time (s)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">130\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Figure 5 shows the time-to-mine difference expressed in seconds between the two approaches, considering new treatment registration requests. On the x-axis we reported the number of treatment registration requests, whereas on the y-axis we reported the processing time expressed in seconds. Looking at the graph, we can observe that for roughly 80 treatment registration requests, both configurations present a similar trend, whereas by increasing the number of requests, the hybrid Ethereum network shows better performances than the public one thanks to the reduced waiting time in the mining queue.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Celesti_Sensors20_20-9.png\" class=\"image wiki-link\" data-key=\"dccc3751af86acf92478964cdfe9a8e7\"><img alt=\"Fig5 Celesti Sensors20 20-9.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/01\/Fig5_Celesti_Sensors20_20-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Time comparison for public and hybrid Ethereum network approaches considering a varying number of treatment registration requests<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Figure 6 describes a cost comparison in Ether (ETH) for the two approaches. On the x-axis we reported the number of treatment registration requests, whereas on the y-axis we reported the cost expressed in Ether (ETH). From our tests, we appreciated that the average cost of a single transaction (i.e., a simple smart contract representing a new treatment memorization or modification) that has to be written in the public Ethereum blockchain is roughly 0.0002 ETH. It can be noted that for a small number of transactions there is not a perceptible convenience in preferring the proposed hybrid approach. This is because at least one transaction per day is written in the public Ethereum blockchain. However, it is clear that cost savings increases exponentially, increasing the number of transactions because only one public transaction out of every 100 private treatments will be paid with ETH cryptocurrency, resulting in an important cost-saving for the FHC ecosystem.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Celesti_Sensors20_20-9.png\" class=\"image wiki-link\" data-key=\"71e454c95049a99904372cf1b1b500df\"><img alt=\"Fig6 Celesti Sensors20 20-9.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/35\/Fig6_Celesti_Sensors20_20-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6.<\/b> Cost comparison for public and hybrid Ethereum network approaches<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Test results demonstrate how the Ethereum hybrid network approach can be adopted to improve both processing time and cost savings, maintaining the same level of accountability and data certification as the public approach through certification on blockchain.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions_and_future_work\">Conclusions and future work<\/span><\/h2>\n<p>This paper demonstrated how an IoT-connected laboratory service can be developed through a healthcare workflow running in an federated hospital cloud environment leveraging blockchain. Experimental results highlight that the performance of the Ethereum hybrid network certification system is improved in terms of cost and response time compared to an alternative public approach.\n<\/p><p>Without a doubt, blockchain technology is destined to evolve, improving system capabilities and robustness, and public test instances with different consensus protocols will be made available with benefits on performance and scalability.\n<\/p><p>In a pandemic-related situation such as that currently caused by the SARS-CoV-2 virus, the need for IoT-connected, cloud-based laboratory services becomes dramatically fundamental to reducing the movement of patients, and by extension reducing the risk of infection in healthcare settings. We hope that with this paper, we succeeded in stimulating the attention of both academic and industrial communities toward the adoption of Blockchain in the healthcare context to speed up the development of innovative services.\n<\/p><p>In the future, this work can be extended by integrating a comprehensive healthcare scenario with different involved organizations, such as a pharmaceutical company that registers in the blockchain all the phases of drug production, all the way to final packaging and shipment. In this case, when a patient buys a prescribed medicine, it would be possible to link the patient with the medicine box, which would mean an important step towards the end of falsified pharmaceuticals and an important assurance for the end user, who can be identified in case a specific drug package has been recalled.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Conceptualization, A.C., M.F., M.V., A.R. (Agata Romano); Methodology, A.R. (Armando Ruggeri), A.G.; Software; Validation, A.R. (Armando Ruggeri), A.G.; Formal Analysis A.R. (Armando Ruggeri); Investigation, A.C., M.F., M.V.; Resources, A.R. (Agata Romano), A.R. (Armando Ruggeri); Data Curation A.R. (Armando Ruggeri); Writing\u2014Original Draft Preparation A.C., A.R. (Agata Romano), A.R. (Armando Ruggeri); Writing\u2014Review & Editing, A.C., A.R. (Armando Ruggeri), M.F., M.V., A.R. (Agata Romano); Visualization, A.R. (Armando Ruggeri), A.G.; Supervision, A.C., M.V.; Project Administration, A.C., M.V.; Funding Acquisition, A.C., M.V. All authors have read and agreed to the published version of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research was funded by the Italian PON project \u201cTALISMAN,\u201d grant ARS01_01116 and by the Italian Healthcare Ministry Young Researcher project \u201cDo Severe acquired brain injury patients benefit from Telerehabilitation? A Cost-effectiveness analysis study,\u201d grant GR-2016-02361306.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HassenteufelTheRole20-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HassenteufelTheRole20_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hassenteufel, P.; Schweyer, F.-X.; Gerlinger, T. et al. (2020). \"The role of professional groups in policy change: Physician's organizations and the issue of local medical provision shortages in France and Germany\". <i>European Policy Analysis<\/i> <b>6<\/b> (1): 38\u201357. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fepa2.1073\" target=\"_blank\">10.1002\/epa2.1073<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+role+of+professional+groups+in+policy+change%3A+Physician%27s+organizations+and+the+issue+of+local+medical+provision+shortages+in+France+and+Germany&rft.jtitle=European+Policy+Analysis&rft.aulast=Hassenteufel%2C+P.%3B+Schweyer%2C+F.-X.%3B+Gerlinger%2C+T.+et+al.&rft.au=Hassenteufel%2C+P.%3B+Schweyer%2C+F.-X.%3B+Gerlinger%2C+T.+et+al.&rft.date=2020&rft.volume=6&rft.issue=1&rft.pages=38%E2%80%9357&rft_id=info:doi\/10.1002%2Fepa2.1073&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Dubas-Jak.C3.B3bczykImpact19-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Dubas-Jak.C3.B3bczykImpact19_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dubas-Jak\u00f3bczyk, K.; Domaga\u0142a, A.; Mikos, M. (2019). \"Impact of the doctor deficit on hospital management in Poland: A mixed-method study\". <i>International Journal of Health Planning and Management<\/i> <b>34<\/b> (1): 187-195. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fhpm.2612\" target=\"_blank\">10.1002\/hpm.2612<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30132977\" target=\"_blank\">30132977<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Impact+of+the+doctor+deficit+on+hospital+management+in+Poland%3A+A+mixed-method+study&rft.jtitle=International+Journal+of+Health+Planning+and+Management&rft.aulast=Dubas-Jak%C3%B3bczyk%2C+K.%3B+Domaga%C5%82a%2C+A.%3B+Mikos%2C+M.&rft.au=Dubas-Jak%C3%B3bczyk%2C+K.%3B+Domaga%C5%82a%2C+A.%3B+Mikos%2C+M.&rft.date=2019&rft.volume=34&rft.issue=1&rft.pages=187-195&rft_id=info:doi\/10.1002%2Fhpm.2612&rft_id=info:pmid\/30132977&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Moreno-VozmedianoBEACON16-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Moreno-VozmedianoBEACON16_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moreno-Vozmediano, R.; Huedo, E.; Llorente, I.M. et al. (2015). \"BEACON: A Cloud Network Federation Framework\". <i>Proceedings of ESOCC 2015: Advances in Service-Oriented and Cloud Computing<\/i>: 325\u201337. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-319-33313-7_25\" target=\"_blank\">10.1007\/978-3-319-33313-7_25<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BEACON%3A+A+Cloud+Network+Federation+Framework&rft.jtitle=Proceedings+of+ESOCC+2015%3A+Advances+in+Service-Oriented+and+Cloud+Computing&rft.aulast=Moreno-Vozmediano%2C+R.%3B+Huedo%2C+E.%3B+Llorente%2C+I.M.+et+al.&rft.au=Moreno-Vozmediano%2C+R.%3B+Huedo%2C+E.%3B+Llorente%2C+I.M.+et+al.&rft.date=2015&rft.pages=325%E2%80%9337&rft_id=info:doi\/10.1007%2F978-3-319-33313-7_25&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CelestiCharact16-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CelestiCharact16_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Celesti, A.; Fazio, M.; Giacobbe, M. et al. (2016). \"Characterizing Cloud Federation in IoT\". <i>Proceedings of the 30th International Conference on Advanced Information Networking and Applications Workshops<\/i>: 93\u201398. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FWAINA.2016.152\" target=\"_blank\">10.1109\/WAINA.2016.152<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Characterizing+Cloud+Federation+in+IoT&rft.jtitle=Proceedings+of+the+30th+International+Conference+on+Advanced+Information+Networking+and+Applications+Workshops&rft.aulast=Celesti%2C+A.%3B+Fazio%2C+M.%3B+Giacobbe%2C+M.+et+al.&rft.au=Celesti%2C+A.%3B+Fazio%2C+M.%3B+Giacobbe%2C+M.+et+al.&rft.date=2016&rft.pages=93%E2%80%9398&rft_id=info:doi\/10.1109%2FWAINA.2016.152&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MulfariHowCloud13-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MulfariHowCloud13_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mulfari, D.; Celesti, A.; Puliafito, A. et al. (2013). \"How cloud computing can support on-demand assistive services\". <i>Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility<\/i>: 1\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2461121.2461140\" target=\"_blank\">10.1145\/2461121.2461140<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+cloud+computing+can+support+on-demand+assistive+services&rft.jtitle=Proceedings+of+the+10th+International+Cross-Disciplinary+Conference+on+Web+Accessibility&rft.aulast=Mulfari%2C+D.%3B+Celesti%2C+A.%3B+Puliafito%2C+A.+et+al.&rft.au=Mulfari%2C+D.%3B+Celesti%2C+A.%3B+Puliafito%2C+A.+et+al.&rft.date=2013&rft.pages=1%E2%80%934&rft_id=info:doi\/10.1145%2F2461121.2461140&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LianAnExplor14-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LianAnExplor14_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lian, J-W.; Yen, D.C.; Wang, Y.-T. (2014). \"An exploratory study to understand the critical factors affecting the decision to adopt cloud computing in Taiwan hospital\". <i>International Journal of Information Management<\/i> <b>34<\/b> (1): 28\u201336. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijinfomgt.2013.09.004\" target=\"_blank\">10.1016\/j.ijinfomgt.2013.09.004<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+exploratory+study+to+understand+the+critical+factors+affecting+the+decision+to+adopt+cloud+computing+in+Taiwan+hospital&rft.jtitle=International+Journal+of+Information+Management&rft.aulast=Lian%2C+J-W.%3B+Yen%2C+D.C.%3B+Wang%2C+Y.-T.&rft.au=Lian%2C+J-W.%3B+Yen%2C+D.C.%3B+Wang%2C+Y.-T.&rft.date=2014&rft.volume=34&rft.issue=1&rft.pages=28%E2%80%9336&rft_id=info:doi\/10.1016%2Fj.ijinfomgt.2013.09.004&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GriggsHealth18-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GriggsHealth18_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Griggs, K.N.; Ossipova, O.; Kohlios, C.P. et al.. \"Healthcare Blockchain System Using Smart Contracts for Secure Automated Remote Patient Monitoring\". <i>Journal of Medical Systems<\/i> <b>42<\/b> (7): 130. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10916-018-0982-x\" target=\"_blank\">10.1007\/s10916-018-0982-x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29876661\" target=\"_blank\">29876661<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Healthcare+Blockchain+System+Using+Smart+Contracts+for+Secure+Automated+Remote+Patient+Monitoring&rft.jtitle=Journal+of+Medical+Systems&rft.aulast=Griggs%2C+K.N.%3B+Ossipova%2C+O.%3B+Kohlios%2C+C.P.+et+al.&rft.au=Griggs%2C+K.N.%3B+Ossipova%2C+O.%3B+Kohlios%2C+C.P.+et+al.&rft.volume=42&rft.issue=7&rft.pages=130&rft_id=info:doi\/10.1007%2Fs10916-018-0982-x&rft_id=info:pmid\/29876661&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZubaydiARev19-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZubaydiARev19_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zubaydi, H.D.; Chong, Y.-W.; Ko, K. et al.. \"A Review on the Role of Blockchain Technology in the Healthcare Domain\". <i>Electronics<\/i> <b>8<\/b> (6): 679. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Felectronics8060679\" target=\"_blank\">10.3390\/electronics8060679<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Review+on+the+Role+of+Blockchain+Technology+in+the+Healthcare+Domain&rft.jtitle=Electronics&rft.aulast=Zubaydi%2C+H.D.%3B+Chong%2C+Y.-W.%3B+Ko%2C+K.+et+al.&rft.au=Zubaydi%2C+H.D.%3B+Chong%2C+Y.-W.%3B+Ko%2C+K.+et+al.&rft.volume=8&rft.issue=6&rft.pages=679&rft_id=info:doi\/10.3390%2Felectronics8060679&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AgboBlock19-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AgboBlock19_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Agbo, C.C.; Mahmoud. Q.H.; Eklund, J.M.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6627742\" target=\"_blank\">\"Blockchain Technology in Healthcare: A Systematic Review\"<\/a>. <i>Healthcare<\/i> <b>7<\/b> (2): 56. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fhealthcare7020056\" target=\"_blank\">10.3390\/healthcare7020056<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6627742\/\" target=\"_blank\">PMC6627742<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30987333\" target=\"_blank\">30987333<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6627742\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6627742<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Blockchain+Technology+in+Healthcare%3A+A+Systematic+Review&rft.jtitle=Healthcare&rft.aulast=Agbo%2C+C.C.%3B+Mahmoud.+Q.H.%3B+Eklund%2C+J.M.&rft.au=Agbo%2C+C.C.%3B+Mahmoud.+Q.H.%3B+Eklund%2C+J.M.&rft.volume=7&rft.issue=2&rft.pages=56&rft_id=info:doi\/10.3390%2Fhealthcare7020056&rft_id=info:pmc\/PMC6627742&rft_id=info:pmid\/30987333&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6627742&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-H.C3.B6lblASys18-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-H.C3.B6lblASys18_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">H\u00f6lbl, M.; Kompara, M.; Kami\u0161ali\u0107, A. et al.. \"A Systematic Review of the Use of Blockchain in Healthcare\". <i>Symmetry<\/i> <b>10<\/b> (10): 470. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fsym10100470\" target=\"_blank\">10.3390\/sym10100470<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Systematic+Review+of+the+Use+of+Blockchain+in+Healthcare&rft.jtitle=Symmetry&rft.aulast=H%C3%B6lbl%2C+M.%3B+Kompara%2C+M.%3B+Kami%C5%A1ali%C4%87%2C+A.+et+al.&rft.au=H%C3%B6lbl%2C+M.%3B+Kompara%2C+M.%3B+Kami%C5%A1ali%C4%87%2C+A.+et+al.&rft.volume=10&rft.issue=10&rft.pages=470&rft_id=info:doi\/10.3390%2Fsym10100470&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-QadriTheFut20-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-QadriTheFut20_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Qadri, Y.A.; Nauman, A.; Zikria, Y.B. et al.. \"The Future of Healthcare Internet of Things: A Survey of Emerging Technologies\". <i>IEEE Communications Surveys & Tutorials<\/i> <b>22<\/b> (2): 1121\u20131167. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCOMST.2020.2973314\" target=\"_blank\">10.1109\/COMST.2020.2973314<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Future+of+Healthcare+Internet+of+Things%3A+A+Survey+of+Emerging+Technologies&rft.jtitle=IEEE+Communications+Surveys+%26+Tutorials&rft.aulast=Qadri%2C+Y.A.%3B+Nauman%2C+A.%3B+Zikria%2C+Y.B.+et+al.&rft.au=Qadri%2C+Y.A.%3B+Nauman%2C+A.%3B+Zikria%2C+Y.B.+et+al.&rft.volume=22&rft.issue=2&rft.pages=1121%E2%80%931167&rft_id=info:doi\/10.1109%2FCOMST.2020.2973314&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChakrabortyASec19-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChakrabortyASec19_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chakraborty, S.; Aich, S.; Jim, H.-C.. \"A Secure Healthcare System Design Framework using Blockchain Technology\". <i>Proceedings of the 21st International Conference on Advanced Communication Technology<\/i>: 1260\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.23919%2FICACT.2019.8701983\" target=\"_blank\">10.23919\/ICACT.2019.8701983<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Secure+Healthcare+System+Design+Framework+using+Blockchain+Technology&rft.jtitle=Proceedings+of+the+21st+International+Conference+on+Advanced+Communication+Technology&rft.aulast=Chakraborty%2C+S.%3B+Aich%2C+S.%3B+Jim%2C+H.-C.&rft.au=Chakraborty%2C+S.%3B+Aich%2C+S.%3B+Jim%2C+H.-C.&rft.pages=1260%E2%80%9364&rft_id=info:doi\/10.23919%2FICACT.2019.8701983&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DasaklisBlock18-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DasaklisBlock18_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dasaklis, T.K.; Casino, F.; Patsakis, C.. \"Blockchain Meets Smart Health: Towards Next Generation Healthcare Services\". <i>Proceedings of the 9th International Conference on Information, Intelligence, Systems and Applications<\/i>: 1\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FIISA.2018.8633601\" target=\"_blank\">10.1109\/IISA.2018.8633601<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Blockchain+Meets+Smart+Health%3A+Towards+Next+Generation+Healthcare+Services&rft.jtitle=Proceedings+of+the+9th+International+Conference+on+Information%2C+Intelligence%2C+Systems+and+Applications&rft.aulast=Dasaklis%2C+T.K.%3B+Casino%2C+F.%3B+Patsakis%2C+C.&rft.au=Dasaklis%2C+T.K.%3B+Casino%2C+F.%3B+Patsakis%2C+C.&rft.pages=1%E2%80%938&rft_id=info:doi\/10.1109%2FIISA.2018.8633601&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SrivastavaALight19-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SrivastavaALight19_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Srivastava, G.; Crichigno, J.; Dhar, S.. \"A Light and Secure Healthcare Blockchain for IoT Medical Devices\". <i>Proceedings of the 2019 IEEE Canadian Conference of Electrical and Computer Engineering<\/i>: 1\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCCECE.2019.8861593\" target=\"_blank\">10.1109\/CCECE.2019.8861593<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Light+and+Secure+Healthcare+Blockchain+for+IoT+Medical+Devices&rft.jtitle=Proceedings+of+the+2019+IEEE+Canadian+Conference+of+Electrical+and+Computer+Engineering&rft.aulast=Srivastava%2C+G.%3B+Crichigno%2C+J.%3B+Dhar%2C+S.&rft.au=Srivastava%2C+G.%3B+Crichigno%2C+J.%3B+Dhar%2C+S.&rft.pages=1%E2%80%935&rft_id=info:doi\/10.1109%2FCCECE.2019.8861593&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HosseinBlock19-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HosseinBlock19_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hossein, K.M.; Esmaeilli, M.E.; Dargahi, T. et al.. \"Blockchain-Based Privacy-Preserving Healthcare Architecture\". <i>Proceedings of the 2019 IEEE Canadian Conference of Electrical and Computer Engineering<\/i>: 1\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCCECE.2019.8861857\" target=\"_blank\">10.1109\/CCECE.2019.8861857<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Blockchain-Based+Privacy-Preserving+Healthcare+Architecture&rft.jtitle=Proceedings+of+the+2019+IEEE+Canadian+Conference+of+Electrical+and+Computer+Engineering&rft.aulast=Hossein%2C+K.M.%3B+Esmaeilli%2C+M.E.%3B+Dargahi%2C+T.+et+al.&rft.au=Hossein%2C+K.M.%3B+Esmaeilli%2C+M.E.%3B+Dargahi%2C+T.+et+al.&rft.pages=1%E2%80%934&rft_id=info:doi\/10.1109%2FCCECE.2019.8861857&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhangFHIR18-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhangFHIR18_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhang, P.; White, J.; Schmidt, D.C. et al.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6082774\" target=\"_blank\">\"FHIRChain: Applying Blockchain to Securely and Scalably Share Clinical Data\"<\/a>. <i>Computational and Structural Biotechnology Journal<\/i> <b>16<\/b>: 267\u201378. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.csbj.2018.07.004\" target=\"_blank\">10.1016\/j.csbj.2018.07.004<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6082774\/\" target=\"_blank\">PMC6082774<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30108685\" target=\"_blank\">30108685<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6082774\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6082774<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=FHIRChain%3A+Applying+Blockchain+to+Securely+and+Scalably+Share+Clinical+Data&rft.jtitle=Computational+and+Structural+Biotechnology+Journal&rft.aulast=Zhang%2C+P.%3B+White%2C+J.%3B+Schmidt%2C+D.C.+et+al.&rft.au=Zhang%2C+P.%3B+White%2C+J.%3B+Schmidt%2C+D.C.+et+al.&rft.volume=16&rft.pages=267%E2%80%9378&rft_id=info:doi\/10.1016%2Fj.csbj.2018.07.004&rft_id=info:pmc\/PMC6082774&rft_id=info:pmid\/30108685&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6082774&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RamaniSecure18-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RamaniSecure18_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-RamaniSecure18_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ramani, V.; Kumar, T.; Bracken, A. et al.. \"Secure and Efficient Data Accessibility in Blockchain Based Healthcare Systems\". <i>Proceedings of the 2018 IEEE Global Communications Conference<\/i>: 206\u201312. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FGLOCOM.2018.8647221\" target=\"_blank\">10.1109\/GLOCOM.2018.8647221<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Secure+and+Efficient+Data+Accessibility+in+Blockchain+Based+Healthcare+Systems&rft.jtitle=Proceedings+of+the+2018+IEEE+Global+Communications+Conference&rft.aulast=Ramani%2C+V.%3B+Kumar%2C+T.%3B+Bracken%2C+A.+et+al.&rft.au=Ramani%2C+V.%3B+Kumar%2C+T.%3B+Bracken%2C+A.+et+al.&rft.pages=206%E2%80%9312&rft_id=info:doi\/10.1109%2FGLOCOM.2018.8647221&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BiggsBlock17-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BiggsBlock17_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Biggs, J.; Hinish, S.R.; Natale, M.A. et al. (2017). \"Blockchain: Revolutionizing the Global Supply Chain by Building Trust and Transparency\". <i>Academia.edu<\/i>. Rutgers University.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Blockchain%3A+Revolutionizing+the+Global+Supply+Chain+by+Building+Trust+and+Transparency&rft.atitle=Academia.edu&rft.aulast=Biggs%2C+J.%3B+Hinish%2C+S.R.%3B+Natale%2C+M.A.+et+al.&rft.au=Biggs%2C+J.%3B+Hinish%2C+S.R.%3B+Natale%2C+M.A.+et+al.&rft.date=2017&rft.pub=Rutgers+University&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShenMedChain19-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShenMedChain19_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shen, B.; Guo, J.; Yang, Y.. \"MedChain: Efficient Healthcare Data Sharing via Blockchain\". <i>Applied Sciences<\/i> <b>9<\/b> (6): 1207. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fapp9061207\" target=\"_blank\">10.3390\/app9061207<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MedChain%3A+Efficient+Healthcare+Data+Sharing+via+Blockchain&rft.jtitle=Applied+Sciences&rft.aulast=Shen%2C+B.%3B+Guo%2C+J.%3B+Yang%2C+Y.&rft.au=Shen%2C+B.%3B+Guo%2C+J.%3B+Yang%2C+Y.&rft.volume=9&rft.issue=6&rft.pages=1207&rft_id=info:doi\/10.3390%2Fapp9061207&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhatoonABlock20-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KhatoonABlock20_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khatoon, A.. \"A Blockchain-Based Smart Contract System for Healthcare Management\". <i>Electronics<\/i> <b>9<\/b> (1): 94. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Felectronics9010094\" target=\"_blank\">10.3390\/electronics9010094<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Blockchain-Based+Smart+Contract+System+for+Healthcare+Management&rft.jtitle=Electronics&rft.aulast=Khatoon%2C+A.&rft.au=Khatoon%2C+A.&rft.volume=9&rft.issue=1&rft.pages=94&rft_id=info:doi\/10.3390%2Felectronics9010094&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SatamrajuProof20-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SatamrajuProof20_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Satamraju, K.P.; Malarkodi, B.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7085612\" target=\"_blank\">\"Proof of Concept of Scalable Integration of Internet of Things and Blockchain in Healthcare\"<\/a>. <i>Sensors<\/i> <b>20<\/b> (5): 1389. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs20051389\" target=\"_blank\">10.3390\/s20051389<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7085612\/\" target=\"_blank\">PMC7085612<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32138380\" target=\"_blank\">32138380<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7085612\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7085612<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Proof+of+Concept+of+Scalable+Integration+of+Internet+of+Things+and+Blockchain+in+Healthcare&rft.jtitle=Sensors&rft.aulast=Satamraju%2C+K.P.%3B+Malarkodi%2C+B.&rft.au=Satamraju%2C+K.P.%3B+Malarkodi%2C+B.&rft.volume=20&rft.issue=5&rft.pages=1389&rft_id=info:doi\/10.3390%2Fs20051389&rft_id=info:pmc\/PMC7085612&rft_id=info:pmid\/32138380&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7085612&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TMCHome-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TMCHome_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.telemedcare.com\/\" target=\"_blank\">\"TeleMedCare\"<\/a>. TeleMedCare<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.telemedcare.com\/\" target=\"_blank\">https:\/\/www.telemedcare.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=TeleMedCare&rft.atitle=&rft.pub=TeleMedCare&rft_id=https%3A%2F%2Fwww.telemedcare.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EverHome-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EverHome_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.eversensediabetes.com\/\" target=\"_blank\">\"Eversense\"<\/a>. Senseonics, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.eversensediabetes.com\/\" target=\"_blank\">https:\/\/www.eversensediabetes.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Eversense&rft.atitle=&rft.pub=Senseonics%2C+Inc&rft_id=https%3A%2F%2Fwww.eversensediabetes.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Med-Care-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Med-Care_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.t4all.it\/portfolio-articoli\/med-care\/\" target=\"_blank\">\"Med-Care\"<\/a>. T4A Srl<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.t4all.it\/portfolio-articoli\/med-care\/\" target=\"_blank\">http:\/\/www.t4all.it\/portfolio-articoli\/med-care\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Med-Care&rft.atitle=&rft.pub=T4A+Srl&rft_id=http%3A%2F%2Fwww.t4all.it%2Fportfolio-articoli%2Fmed-care%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HemoScreen-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HemoScreen_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.pixcell-medical.com\/hemoscreen\/\" target=\"_blank\">\"HemoScreen\"<\/a>. PixCell Medical Technologies Ltd<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.pixcell-medical.com\/hemoscreen\/\" target=\"_blank\">https:\/\/www.pixcell-medical.com\/hemoscreen\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 March 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=HemoScreen&rft.atitle=&rft.pub=PixCell+Medical+Technologies+Ltd&rft_id=https%3A%2F%2Fwww.pixcell-medical.com%2Fhemoscreen%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LabGeoPT10-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LabGeoPT10_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20200505231514\/https:\/\/samsunghealthcare.com\/en\/products\/InVitroDiagnostics\/Samsung%20LABGEO%20PT10\/Point-of-Care\/benefit\" target=\"_blank\">\"LabGeo PT10\"<\/a>. Samsung. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/samsunghealthcare.com\/en\/products\/InVitroDiagnostics\/Samsung%20LABGEO%20PT10\/Point-of-Care\/benefit\" target=\"_blank\">the original<\/a> on 05 May 2020<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20200505231514\/https:\/\/samsunghealthcare.com\/en\/products\/InVitroDiagnostics\/Samsung%20LABGEO%20PT10\/Point-of-Care\/benefit\" target=\"_blank\">https:\/\/web.archive.org\/web\/20200505231514\/https:\/\/samsunghealthcare.com\/en\/products\/InVitroDiagnostics\/Samsung%20LABGEO%20PT10\/Point-of-Care\/benefit<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=LabGeo+PT10&rft.atitle=&rft.pub=Samsung&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20200505231514%2Fhttps%3A%2F%2Fsamsunghealthcare.com%2Fen%2Fproducts%2FInVitroDiagnostics%2FSamsung%2520LABGEO%2520PT10%2FPoint-of-Care%2Fbenefit&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GallettaAnApp17-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GallettaAnApp17_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Galletta, A.; Bonanno, L.; Celesti, A. et al.. \"An approach to share MRI data over the Cloud preserving patients' privacy\". <i>Proceedings of the 2017 IEEE Symposium on Computers and Communications<\/i>: 94\u201399. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FISCC.2017.8024511\" target=\"_blank\">10.1109\/ISCC.2017.8024511<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+approach+to+share+MRI+data+over+the+Cloud+preserving+patients%27+privacy&rft.jtitle=Proceedings+of+the+2017+IEEE+Symposium+on+Computers+and+Communications&rft.aulast=Galletta%2C+A.%3B+Bonanno%2C+L.%3B+Celesti%2C+A.+et+al.&rft.au=Galletta%2C+A.%3B+Bonanno%2C+L.%3B+Celesti%2C+A.+et+al.&rft.pages=94%E2%80%9399&rft_id=info:doi\/10.1109%2FISCC.2017.8024511&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GallettaBigMRI17-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GallettaBigMRI17_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Galletta, A.; Celesti, A.; Tusa, F. et al.. \"Big MRI Data Dissemination and Retrieval in a Multi-Cloud Hospital Storage System\". <i>Proceedings of the 2017 International Conference on Digital Health<\/i>: 162\u201366. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3079452.3079507\" target=\"_blank\">10.1145\/3079452.3079507<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Big+MRI+Data+Dissemination+and+Retrieval+in+a+Multi-Cloud+Hospital+Storage+System&rft.jtitle=Proceedings+of+the+2017+International+Conference+on+Digital+Health&rft.aulast=Galletta%2C+A.%3B+Celesti%2C+A.%3B+Tusa%2C+F.+et+al.&rft.au=Galletta%2C+A.%3B+Celesti%2C+A.%3B+Tusa%2C+F.+et+al.&rft.pages=162%E2%80%9366&rft_id=info:doi\/10.1145%2F3079452.3079507&rfr_id=info:sid\/en.wikipedia.org:Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. The original title\u2014<i>Blockchain-Based Healthcare Workflow for Tele-Medical Laboratory in Federated Hospital IoT Clouds<\/i>\u2014was changed slightly as \"tele-medical laboratory\" is a confusing phrase that elicits thoughts of \"telemedicine,\" which is not an accurate portrayal of the article. The new title intends to capture the true spirit of the concept, and this change was also propogated throughout this version of the article. In some cases important information was missing from the references, and that information was added. The Samsung Labgeo PT10S has since been discontinued; an archived version of the URL for it is used here.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150604\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.611 seconds\nReal time usage: 5.501 seconds\nPreprocessor visited node count: 21095\/1000000\nPreprocessor generated node count: 32904\/1000000\nPost\u2010expand include size: 147115\/2097152 bytes\nTemplate argument size: 51931\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 430.597 1 - -total\n 85.43% 367.840 1 - Template:Reflist\n 72.64% 312.768 28 - Template:Citation\/core\n 62.65% 269.760 22 - Template:Cite_journal\n 13.61% 58.584 6 - Template:Cite_web\n 8.92% 38.430 1 - Template:Infobox_journal_article\n 8.48% 36.521 1 - Template:Infobox\n 6.72% 28.934 30 - Template:Citation\/identifier\n 5.73% 24.659 80 - Template:Infobox\/row\n 3.90% 16.791 30 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12519-0!*!0!!en!5!* and timestamp 20210705150559 and revision id 42670\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds\">https:\/\/www.limswiki.org\/index.php\/Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","e49209f2666c8e463be219dc620f0ec5_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/e\/e6\/Fig1_Celesti_Sensors20_20-9.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/57\/Fig2_Celesti_Sensors20_20-9.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/c\/c8\/Fig3_Celesti_Sensors20_20-9.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/f\/f3\/Fig4_Celesti_Sensors20_20-9.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/0\/01\/Fig5_Celesti_Sensors20_20-9.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/35\/Fig6_Celesti_Sensors20_20-9.png"],"e49209f2666c8e463be219dc620f0ec5_timestamp":1625497559,"7ef870605af2fae6ca37ea1b35850e50_type":"article","7ef870605af2fae6ca37ea1b35850e50_title":"Popularity and performance of bioinformatics software: The case of gene set analysis (Xie et al. 2021)","7ef870605af2fae6ca37ea1b35850e50_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis","7ef870605af2fae6ca37ea1b35850e50_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Popularity and performance of bioinformatics software: The case of gene set analysis\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nPopularity and performance of bioinformatics software: The case of gene set analysisJournal\n \nBMC BioinformaticsAuthor(s)\n \nXie, Chengshu; Jauhari, Shaurya; Mora, AntonioAuthor affiliation(s)\n \nGuangzhou Medical University, Guangzhou Institutes of Biomedicine and HealthPrimary contact\n \nOnline formYear published\n \n2021Volume and issue\n \n22Article #\n \n191DOI\n \n10.1186\/s12859-021-04124-5ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-021-04124-5Download\n \nhttps:\/\/bmcbioinformatics.biomedcentral.com\/track\/pdf\/10.1186\/s12859-021-04124-5.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Results \n\n3.1 Popularity \n3.2 Performance \n3.3 Popularity versus performance \n3.4 Performance tools \n\n\n4 Discussion \n5 Conclusions \n6 Methods \n\n6.1 Definitions \n6.2 Construction of the database \n6.3 Descriptive statistics \n6.4 Popularity rankings \n6.5 Performance study \n6.6 Construction of performance-measuring software \n\n\n7 Supplementary information \n8 Acknowledgements \n\n8.1 Contributions \n8.2 Availability of data and materials \n8.3 Funding \n8.4 Competing interests \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nBackground: Gene set analysis (GSA) is arguably the method of choice for the functional interpretation of omics results. This work explores the popularity and the performance of all the GSA methodologies and software published during the 20 years since its inception. \"Popularity\" is estimated according to each paper's citation counts, while \"performance\" is based on a comprehensive evaluation of the validation strategies used by papers in the field, as well as the consolidated results from the existing benchmark studies.\nResults: Regarding popularity, data is collected into an online open database (\"GSARefDB\") which allows browsing bibliographic and method-descriptive information from 503 GSA paper references; regarding performance, we introduce a repository of Jupyter Notebook workflows and Shiny apps for automated benchmarking of GSA methods (\u201cGSA-BenchmarKING\u201d). After comparing popularity versus performance, results show discrepancies between the most popular and the best performing GSA methods.\nConclusions: The above-mentioned results call our attention towards the nature of the tool selection procedures followed by researchers and raise doubts regarding the quality of the functional interpretation of biological datasets in current biomedical studies. Suggestions for the future of the functional interpretation field are made, including strategies for education and discussion of GSA tools, better validation and benchmarking practices, reproducibility, and functional re-analysis of previously reported data.\nKeywords: bioinformatics, pathway analysis, gene set analysis, benchmark, GSEA\n\nBackground \nBioinformatics method and software selection is an important problem in biomedical research, due to the possible consequences of choosing the wrong methods among the existing myriad of computational methods and software available. Errors in software selection may include the use of outdated or suboptimal methods (or reference databases) or misunderstanding the parameters and assumptions behind the chosen methods. Such errors may affect the conclusions of the entire research project and nullify the efforts made on the rest of the experimental and computational pipeline.[1]\nThis paper discusses two main factors that motivate researchers to make method or software choices, that is, the popularity (defined as the perceived frequency of use of a tool among members of the community) and the performance (defined as a quantitative quality indicator measured and compared to alternative tools). This study is focused on the field of \u201cgene set analysis\u201d (GSA), where the popularity and performance of bioinformatics software show discrepancies, and therefore the question appears whether biomedical sciences have been using the best available GSA methods or not.\nGSA is arguably the most common procedure for functional interpretation of omics data, and, for the purposes of this paper, we define it as the comparison of a query gene set (a list or a rank of differentially expressed genes, for example) to a reference database, using a particular statistical method, in order to interpret it as a rank of significant pathways, functionally related gene sets, or ontology terms. Such definition includes the categories that have been traditionally called \"gene set analysis,\" \"pathway analysis,\" \"ontology analysis,\" and \"enrichment analysis.\" All GSA methods have a common goal, which is the interpretation of biomolecular data in terms of annotated gene sets, while they differ depending on characteristics of the computational approach (for more details, see the \"Methods\" section, as well as Fig. 1 of Mora[2]). \nGSA has arrived to 20 years of existence since the original paper of Tavazoie et al.[3], and many statistical methods and software tools have been developed during this time. A popular review paper listed 68 GSA tools[4], while a second review reported an additional 33 tools[5], and a third paper 22 tools.[6] We have built the most comprehensive list of references to date (503 papers), and we have quantified each paper\u2019s influence according to their current number of citations (see Additional file 1 and Mora's GSARefDB[7]). The most common GSA methods include Over-Representation Analysis (ORA), such as that found with DAVID[8]; Functional Class Scoring (FCS), such as that found with GSEA[9]; and Pathway-Topology-based (PT) methods, such as that found with SPIA.[10] All been extensively reviewed. In order to know more about them, the reader may consult any of the 62 published reviews documented in Additional file 1. We have also recently reviewed other types of GSA methods.[2]\nThe first part of the analysis is a study on GSA method and software popularity based on a comprehensive database of 503 papers for all methods, tools, platforms, reviews, and benchmarks of the GSA field, collectively cited 134,222 times between 1999 and 2019, including their popularity indicators and other relevant characteristics. The second part is a study on performance based on the validation procedures reported in the papers, introducing new methods together with all the existing independent benchmark studies in the above-mentioned database. Instead of recommending one single GSA method, we focus on discussing better benchmarking practices and generating benchmarking tools that follow such practices. Together, both parts of the study allow us to compare the popularity and performance of GSA tools but also to explore possible explanations for the popularity phenomenon and the problems that limit the execution and adoption of independent performance studies. In the end, some practices are suggested to guarantee that bioinformatics software selection is guided by the most appropriate metrics.\n\nResults \nPopularity \nThe number of paper citations has been used as a simple (yet imperfect) measure of a GSA method\u2019s popularity. We collected 350 references to GSA papers of methods, software or platforms (Additional file 1: Tab 1), as well as 91 references to GSA papers for non-mRNA omics tools (Additional file 1: Tabs 3, 4), and 62 GSA reviews or benchmark studies (Additional file 1: Tabs 2, 3, 4), which have been organized into a manually curated open database (GSARefDB). GSARefDB can be either downloaded or accessed through a Shiny interface. Figure 1 summarizes some relevant information from the database (GSARefDB v.1.0). \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Statistics from GSARefDB v.1.0 (for more recent statistics, visit our website). a Number of GSA publications per year. b Number of publications per type of GSA method. c Number of publications per used programming language. d Number of citations per used programming language. e Website availability. f Number of publications per reported validation method.\n\n\n\nThe citation count shows that the most influential GSA method in history is Gene Set Enrichment Analysis (GSEA), published in 2003, with more than twice as many citations as its follower, the ORA platform called DAVID (17,877 versus 7,500 citations). In general, the database shows that the field contains a few extremely popular papers and many papers with low popularity. Figure 1b shows that the list of tools is mainly composed of ORA and FCS methods, while the newer and lesser known PT and Network Interaction (NI) methods are less common (and generally found at the bottom of the popularity rank).\nIt could be hypothesized that the popularity of a GSA tool does not always depend on being the best for that particular project, and it could be related to variables such as its user-friendliness instead. The database allows us to compute citations-per-programming-language, which we use as an approximation to friendliness. Figure 1c shows that the majority of the GSA papers correspond to R tools, but, in spite of that, Fig. 1d shows that most of the citations correspond to web platforms followed by stand-alone applications, which are friendlier to users. Worth mentioning, the last column of the database shows that there are a large number of tools that are not maintained anymore, with broken web links to tools or databases, which makes their evaluation impossible. This phenomenon is a common bioinformatics problem that has also been reported elsewhere.[11] Figure 1e shows that one-third of the reported links in GSA papers are now broken links.\nBesides ranking papers according to their all-time popularity, a current-popularity rank was also built (Additional file 1: Tab 5). To accomplish that, a version of the database generated in May 2018 was compared to a version of the database generated in April 2019. The \"current popularity\" rank revealed the same trends as the \"overall\" rank. GSEA is still, by far, the most popular method. The other tools that are being currently cited are clusterProfiler[12], Enrichr[13], GOseq[14], DAVID[8], and ClueGO[15], followed by GOrilla, KOBAS, BiNGO, ToppGene, GSVA, WEGO, agriGO, and WebGestalt. ORA and FCS methods are still the most popular ones, with 3534 combined citations for all ORA methods and 2185 citations for all FCS methods, while PT and NI methods have 111 and 50 combined citations respectively. In contrast, single-sample methods have 278 combined citations, while time-course methods have 67. Regarding reviews, an extremely popular paper from 2009[4] is still currently the most popular, even though it doesn\u2019t take into account the achievements of the last 10 years.\n\nPerformance \nThe subject of validation of bioinformatics software deserves more attention.[16] A review of the scientific validation approaches followed by the top 153 GSA tool papers in the database (Additional file 1: Tab 6) found multiple validation strategies that were classified into 19 categories. 61 out of the 153 papers include a validation procedure, and the most commonly found validation strategy is \u201cConsistency with biological knowledge,\u201d defined as the fact that our method\u2019s results explain the knowledge in the field better than the rival methods (which is commonly accomplished through a literature search). Other common strategies (though less common) are the comparisons of the number of hits, classification accuracy, and consistency of results between similar samples. Important strategies, such as comparing statistical power, benchmark studies, and simulations, are less used. The least used strategies include experimental confirmation of predictions and semi-blind procedures where a person collects samples and another person applies the tool to guess tissue or condition. Our results have been summarized in Fig. 1f (above) and Additional file 1: Tab 6. We can see that the frequency of use of the above-mentioned validation strategies is inversely proportional to their reliability. For example, commonly used strategies such as \u201cconsistency with biological knowledge\u201d can be subjective, and comparing the number of hits of our method with other methods on a Venn diagram[17] is a measure of agreement, not truth. On the other hand, the least used strategies, such as experimental confirmation or benchmark and simulation studies, are the better alternatives.\nThe next step of our performance study was a review of all the independent benchmark and simulation studies existing in the GSA field, whose references are collected in Additional file 1: Tab 2. Table 1 summarizes 10 benchmark studies of GSA methods, with different sizes, scopes, and method recommendations. A detailed description and discussion of each of the benchmarking studies can be found in Additional file 2. The sizes of all benchmarking studies are small when compared to the amount of existing methods that we have mentioned before, while their lists of best performing methods show little overlap. Only a few methods are mentioned as best performers in more than one study, including ORA methods (such as hypergeometric)[3], FCS methods (such as PADOG)[18], SS methods (such as PLAGE)[19], and PT methods (such as SPIA\/ROntoTools).[20]\n\n\n\n\n\n\n\nTable 1. Ten benchmark studies of GSA methods, from 2012 to 2020, showing a plurality of scopes, sizes, and method recommendations. Details on each study can be found in Additional file 2. ORA, over-representation analysis; FCS, functional class scoring; PT, pathway topology-based; SS, single-sample; NI, network interaction.\n\n\nStudy reference\n\nScope\n\nSize\n\nCriteria\n\nBest performing methods\n\n\nNaeem et al.[21]\n\nORA and FCS methods\n\n14 methods\n\nMethod\u2019s AUC (evaluated by predicting targets of TFs and miRNAs)\n\nANOVA, Z-SCORE, and Wilcoxon\u2019s rank sum (WRS)\n\n\nTarca et al.[22]\n\nORA, FCS, and SS methods\n\n16 methods\n\nPrioritization, Sensitivity, and FPR\n\nGLOBALTEST and PLAGE (sensitivity), PADOG and ORA (prioritization), and CAMERA (FPR). Author\u2019s general recommendation: PLAGE, GLOBALTEST, and PADOG.\n\n\nBayerlov\u00e1 et al.[23]\n\nORA, FCS, and PT methods\n\n7 methods\n\nSensitivity and prioritization (for benchmark), and Sensitivity, specificity, and accuracy (for simulations of pathway overlap)\n\nFor benchmark: CePaGSA (sensitivity) and PathNet (prioritization). For simulation -original pathways: CePAGSA (sensitivity), WRS (specificity), and WRS (accuracy). For simulation -non-overlapping pathways: KS (sensitivity), and SPIA, CePaORA, CePaGSA, and PathNet (specificity and accuracy).\n\n\nJaakkola et al.[24]\n\nORA, FCS, and PT methods\n\n5 methods\n\nConsistency of significant pathways between datasets, and Sensitivity\n\nSPIA and CePaORA (consistency), and SPIA, CePaORA, and NetGSA (sensitivity). Author\u2019s general recommendation: SPIA.\n\n\nDe Meyer[25]\n\nORA, FCS, and NI methods\n\n4 methods\n\nPrioritization, Sensitivity, and Specificity\n\nPADOG (specificity) and BinoX (sensitivity).\n\n\nLim et al.[26]\n\nSS\/Pathway-activity methods\n\n13 methods\n\nClassification performance, preservation of data structure, robustness to noise, and reproducibility between pathway databases\n\nESEA, Pathifier, SAS, and PADOG (classification tasks), Pathifier and PLAGE (data structure), ssGSEA (robustness), and individPath, Pathifier, and SAS (reproducibility). Author\u2019s general recommendation: Pathifier, SAS, and individPath.\n\n\nNguyen et al.[27]\n\nORA, FCS, and PT methods\n\n13 methods\n\nIn order of importance: Number of biased pathways, Prioritization, Method\u2019s AUC, and sensitivity (evaluated using both disease target pathways and KO data)\n\nGSEA (bias), PADOG (prioritization), ROntoTools (AUC), and CePaGSA (p-values). Author\u2019s general recommendation: ROntoTools.\n\n\nMa et al.[28]\n\nFCS, PT, and NI methods\n\n9 methods\n\nRanking of empirical powers\n\nDEGraph, followed by PathNet and NetGSA\n\n\nZyla et al.[29]\n\nORA, FCS, and SS methods\n\n9 methods\n\nSensitivity, FPR, prioritization, computational time, and reproducibility\n\nPLAGE (sensitivity), ORA and PADOG (specificity\/FPR), PADOG (prioritization), and CERNO (reproducibility).\n\n\nGeistlinger et al.[30]\n\nORA, FCS, and SS methods\n\n10 methods\n\nSensitivity, computational time, and phenotype relevance score\n\nAuthor\u2019s general recommendation: ROAST and GSVA (for self-contained hypothesis). ORA and PADOG (for competitive hypothesis).\n\n\n\nIt is crucial to notice that the previous studies just covered a small fraction of the entire universe of GSA methods. Also, that there is little overlap between the sets of methods included in different studies, but nevertheless we can still find inconsistencies between results, such as the GSVA, Pathifier, and hypergeometric methods, which are reported both as best performer and as poor performer in different benchmarks. From all the above-mentioned high-performance tools, only \u201cORA\u201d appears in the top 20 currently most popular tools (Additional file 1: Tab 5), suggesting that there is a divorce between popularity and performance (see also Additional file 2: Table 2).\n\nPopularity versus performance \nIn general terms, it is evident that performance studies are still few, small, inconsistent, and dependent on the quality of the benchmarks; however, they tend to recommend tools different to the popular and friendly ones.\nGSEA is one of the most important landmarks in GSA history and the most striking example of the contradictions between popularity and performance metrics of GSA tools. With one exception, none of the recent benchmarks under analysis has reported GSEA to be the best performing method; however, GSEA is still both the all-time and the currently most used tool. Besides that, numerous methods (most of them not included in the previous benchmarks) report that they specifically outperform GSEA in at least one of many possible ways; such methods are highlighted on Additional file 1: Tab 7 with the tag \"COMPARED\". In addition, there have been plenty of developments to the GSEA method itself; for example, alternative functions to score the gene set, alternative options to the permutation step to finding p-values, or using GSEA as part of an extended method (see methods at Additional file 1: Tab 7, with the tag \"PART OF METHOD\"), which reportedly outperform the original GSEA. We have identified a total of 79 papers in those categories (Additional file 1: Tab 7). However, in spite of that, GSEA's overwhelming popularity compared to any other method would suggest otherwise. We can also verify that, with the exception of ORA, most of the above-mentioned high-performing methods tend to occupy lower places in the current popularity ranking (Additional file 2: Table 2).\nIt is hard to estimate how many conclusions from how many articles should be rectified if we rigorously apply our current knowledge on GSA and use the best performing methods in every paper that ever used GSA. However, taking into account that our database (Additional file 1) registers 503 GSA software and tool papers that have been collectively cited 134,222 times, there is a valid concern regarding the need to pay more attention to both the method performance studies and the performance-based tool selection in GSA. Otherwise, the last step of the omics data analysis process may undermine the efforts on the rest of the pipeline.\n\nPerformance tools \nThe previous observation regarding the difference between the most popular and the best performing GSA methods could be incorrect, given that existing benchmarks have followed different methodologies and the proper way to benchmark GSA methods is a discussion in itself. Therefore, we have also reviewed and discussed the existing benchmarking and simulation strategies, in order to extract a few ideas regarding good benchmark practices.\nRigorous benchmark studies are not a straightforward task, given that there is no accepted gold standard to be used in a comparison of GSA tools.[25] Reviewed here are three strategies to build such a gold standard. The first strategy is to apply several different enrichment tools to the same gene sets and use the intersection of the results as a gold-standard. One example of this is EnrichNet[31], which obtains a \u201cset of high confidence benchmark pathways\u201d as the intersection between the top 100 ranks generated by the SAM-GS and GAGE methods. Such a procedure is questionable, as the chosen methods are far from being considered the best (as seen before), and the procedure is more related to agreement than truth. The second strategy is the use of disease datasets clearly associated with one pathway as a gold-standard. For example, Tarca et al.[22] compiled 42 microarray datasets with both healthy and disease-associated samples, where the disease was associated with a KEGG pathway (and, therefore, such \u201ctarget pathway\u201d should be found significant).[22] The third strategy is to leverage knowledge on gene regulation effects. Some authors have used gene sets built from the known targets of specific transcription factors and miRNAs, and then try to predict the changes after over-expression or deletion of such regulators[21], while some others have used datasets with mouse KOs: Pathways containing the KO gene were considered as target pathways, while the others were considered as negatives.[27] Geistlinger et al.[30] have recently introduced a modification of such approach in which they not only look at \u201ctarget pathways\u201d matching the original diseases, but instead create a \u201cgene set relevance ranking\u201d for each disease. To build such rankings, the authors used MalaCards gene scores for disease relevance, which are based on both experimental and bibliometric evidence; then, they used the gene scores to build combined gene set relevance scores for all GO and KEGG terms (using the GeneAnalytics tool). As a result, instead of benchmarking against some \u201ctarget pathways\u201d, the benchmark is done against a \u201cpathway relevance ranking\u201d per disease. Table 2 assesses different performance measurement approaches according to their objectivity, reproducibility, and scalability.\n\n\n\n\n\n\n\nTable 2. Objectivity, Reproducibility, and Scalability of the main performance criteria. Objectivity refers to the results not depending on human interpretation. Reproducibility refers to any researcher being able to find the same results by following the same procedures. Scalability refers to the possibility of easily applying the procedure to an increasingly higher number of methods and datasets.\n\n\nCriteria\n\nObjectivity\n\nReproducibility\n\nScalability\n\nDrawbacks\n\n\nTool agreement\n\nLow\n\nLow\n\nHigh\n\nSubjective\n\n\nConsistency between similar samples\n\nLow\n\nHigh\n\nHigh\n\nSubjective\n\n\nConsistency with biological knowledge (Literature search)\n\nLow\n\nLow\n\nLow\n\nSubjective\n\n\nBenchmark (target pathways as gold standard)\n\nHigh\n\nHigh\n\nHigh\n\nCentered on true positives\n\n\nBenchmark (pathway relevance ranking as gold standard)\n\nHigh\n\nHigh\n\nHigh\n\n-\n\n\nBenchmark (KO\/perturbation data as gold standard)\n\nHigh\n\nHigh\n\nHigh\n\n-\n\n\nSimulations\n\nLow\n\nLow\n\nHigh\n\nHuman-designed datasets may be unrealistic\n\n\n\nA second issue is the selection of the benchmark metrics that quantitatively determine who is the best performer. Well-known metrics such as sensitivity, specificity, precision, or the area under the ROC curve, have been traditionally used. Tarca et al.[22] introduced the use of sensitivity, specificity, and \u201cprioritization,\u201d together with False Positive Rate (FPR), where prioritization is a concept related to the rank of the target pathways for a given method. Zyla et al.[29] have recently extended Tarca's work and recommended using a set of five different metrics: Sensitivity, False positive rate (FPR), Prioritization, Computational time, and Reproducibility. In their approach, reproducibility assigned a high score to methods that showed similar results for different datasets coming from the same condition\/disease and the same technical platform, but different authors\/labs.[29] The chosen metrics are fundamental because no method is the best under all metrics, and every user should start by selecting a method according to the goals of their study, which may need a very sensitive method, a very specific method, or any other of the above-mentioned properties. As a consequence, benchmark studies must be clear regarding the metrics under which they are ranking the method performance[22], and their set of metrics should at least include the previously mentioned ones. Also, it makes no sense to declare success because a new method has better sensitivity than other methods that are known for a poor sensitivity; therefore, it is a logical consequence that any new methods must compare their sensitivity to sensitive methods, their specificity to specific methods, and so on.\nA third issue is that validation procedures should not allow the authors to subjectively choose the methods that the authors want to compare to their new method, as it would be possible for the authors to choose those methods that they can outperform. One alternative to that is using well-established, independent, and comprehensive benchmark studies as a reference; then, when a new method appears, validation should be done by comparison to the top methods from such independent benchmarks. This practice is not common; as an exception, the authors of LEGO[32] explicitly use the top five methods in Tarca et al.'s benchmark.[22]\nAs a final thought, ensembles of methods have been suggested to give better results than any of the single methods.[33] That implies that benchmark studies should not be limited to single tools but include comparisons to ensembles of tools as well. This approach has been followed by at least one benchmark study.[21]\nThe previous analysis makes us think that future GSA benchmarks should include the most recent developments on benchmark theory, as well as be performed on more GSA methods, in order to extract more meaningful conclusions. In order to do this, we have also created a benchmarking platform, \u201cGSA-BenchmarKING,\u201d which is a repository of apps\/workflows\/pipelines that follow the above-mentioned good benchmarking practices and allow benchmarking of GSA software in an easy and automated way. Currently, GSA-BenchmarKING contains Jupyter Notebooks with full workflows for benchmarking GSA methods, and Shiny apps that allow benchmarking with the click of a few buttons. The initial benchmarking tools added to the repository were created by our students as an example, following the previous guidelines, but they are open software and can be permanently improved. The initial tools are focused on two types of methods: single-sample GSA (such as GSVA, Pathifier, SSGSEA, PLAGE, ZSCORE) and genomic-region GSA (such as GREAT, ChIP-Enrich, BroadEnrich, Enrichr, Seq2Pathways). All apps allow the user to define different gold standard datasets (or use ours), select the GSA methods to compare, and select the comparison metrics to plot. In addition, it is possible to keep adding more GSA methods to the app with the help of the community, as each app includes instructions for programmers interested in adding new methods. Given the large amount of GSA methods, more benchmarking tools are needed and welcomed.\n\nDiscussion \nThe GSARefDB has been used to make some initial data exploration on the relationship between popularity and performance of GSA tools. Besides the observations highlighted in this article, such a database can also be the source for more in-depth research. For example, some limitations of our approach include (i) that some papers describe more than one method or platform, and (ii) that some methods are only cited for comparison purposes. We observe that such problems are the exception and not the rule, but future studies might want to take them into account and GSARefDB can still be used as a data source for it.\nIn general terms, it is evident that performance studies are still few, small, inconsistent, and dependent on the quality of the benchmarks; however, they tend to recommend tools different to the popular and friendly ones. Among the possible reasons for the discrepancies between popularity and performance, it has been suggested that software selection may not be entirely related to performance but to factors such as a preference for user-friendly platforms or user-friendly concepts or plots (for example, GSEA\u2019s \u201cenrichment plot\u201d). Also, due to the fact that the objective evaluation of the performance of the different GSA methods is a complicated and time-consuming issue, or to the fact that new methods need more time to be accepted. In network analysis, popularity can be explained by the \u201crich-gets-richer\u201d (popular-gets-more-popular) effect. Using concepts of the \u201cconsumer behavior\u201d field, software selection can be studied as the choice of a popular brand, that is, variables such as confidence in experience (the respect to the researcher\/institution associated to the software), social acceptance and personal image (following the software that everyone else is using), or consumer loyalty (after some time, we are attached to our software and not interested to change).[34] In a recent book, Barabasi[35] suggested that popularity and quality usually go together in situations where performance is clearly measurable. Otherwise, popularity can't be equated to quality. Our study agrees with such an idea. We have found that the most popular GSA software is different from the best performing GSA software, and that thorough performance evaluation is still a pending assignment for bioinformaticians.\nHowever, in the specific case of GSA, there is a complementary explanatory hypothesis: Besides all the progress in GSA theory leading to better-performing methods, at the end of the pipeline, the users usually extract the evidence that they consider relevant from the resulting rank of gene sets, choosing the pathways or GO terms of their interest and ignoring the rest of the ranking, and, therefore, any differences between methods regarding lower p-values, prioritization order, and so on, become less important. It is said that in the step of gene set selection from the final gene set ranking, researchers bring \"context\" to the results but, this way, their subjectivity may be projected to the study. One way out of this is to stimulate the research on \u201ccontext-based GSA.\u201d For example, a recent effort called contextTRAP[36] combines an impact score (from pathway analysis) with a context score (from text mining information that supports that the pathway is relevant to the context of the experiment). Bayesian approximations to GSA using text mining data as context need to be more developed, as any other method that studies the final gene set ranking as a whole.\n\nConclusions \nGiven that popular methods are not necessarily the best, bioinformatics software users should not only be guided by popularity but mainly by performance studies. However, performance studies, in time, must be guided by the general guidelines that we have discussed, that is, researchers should only follow the most convincing benchmark procedures. Such strict recommendations are problematic because performance studies are few, low-coverage, and have a variable quality; therefore, we need more open tools to dynamically review popularity and performance of bioinformatics software, such as those introduced here.\nBased on the previous results and discussion, we argue that the functional interpretation field would benefit from:\n1. having more information and discussion regarding the nature and scope of the existing functional interpretation methods of omics data, as well as more teaching of new and sophisticated methods in bioinformatics courses, more guidelines for selection of a tool, and more popularization of the functional interpretation shortcomings. Also, deeper discussions of the selection of GSA methods for all biomedical papers should be requested.\n2. the permanent evaluation of GSA methods, including better gold standards, an increasing number of comprehensive comparison studies, and better benchmarking practices. The fact that two prestigious computational biology journals have recently created a special edition and a collection specialized on benchmarks are welcome steps in that direction.[37][38]\n3. paying attention to reproducibility and offering open code in code-sharing platforms (such as GitHub), containers with the specific software and library versions used in their work (such as Docker), notebooks (such as RStudio and Jupyter) including scripts with a detailed explanation of their methods, and other strategies to allow reproducibility.\n4. the creation of a culture of functional re-analysis of existing data using new GSA methods, as well as the computational tools to functionally re-analyze existing omics datasets in a streamlined manner.\n5. more rigorous validation procedures for GSA tools. Also, bioinformaticians should get as much training on scientific validation methods and tools as they get on using and building bioinformatics tools.\nProper tool selection is fundamental for generating high-quality results in all scientific fields. This paper suggests that tool performance and tool selection studies, via the popularity-performance evaluation based on an exhaustive reference database, is a methodology that should be followed up, to keep track of the evolution of the tool selection issues in a scientific field. We have also introduced examples of popularity- and performance-measuring software that could help make such studies easier. The reader is here invited to keep following our work for the GSA field, including GSARefDB and GSA-BenchmarKING.\n\nMethods \nDefinitions \nThe concepts involved in this study have been defined in several different ways in the literature. For example, the field under study has been called \"pathway analysis,\"[5] \"enrichment analysis,\"[23] \"gene set analysis,\"[2] \"functional enrichment analysis,\"[12] \u201cgene-annotation enrichment analysis,\u201d[4] and other terms, by different authors. At the same time, the term \"gene set analysis\" has been used to describe the entire field[2], or just the group of ORA and FCS methods (in opposition to methods including pathway or network topology)[23], or even one specific tool.[39] Finally, the term \u201cpathway analysis\u201d has also been used to both describe the entire field[5], or just the group of methods that include pathway topology.[23] For this reason, we have added the following summary of the definitions used in this study for the above-mentioned and other relevant terms.\nbenchmark study: A systematic comparison between computational methods, in which all of them are applied to a gold standard dataset and the success of their gene set predictions are summarized in terms of quantitative metrics (such as sensitivity, specificity, and others).\nfunctional class scoring (FCS): A subset of GSA methods in which the values of a gene-level statistic for all genes in the experiment are aggregated into a gene-set-level statistics[5], and gene set enrichment is computed in terms of the significance of such gene-set-level statistic. FCS methods start from a quantitative ranking of the gene-level statistic for all genes under analysis (in contrast to ORA, which only uses a list of differentially expressed genes). Some popular FCS methods find if the relative position of a gene set in the ranking of all genes is shifted to the top or the bottom of the ranking. For example, the WRS test compares the distribution of ranks of the genes in a gene set to the distribution of ranks of the genes in the complement to the gene set, while the KS test compares the ranks of genes in a gene set to a uniform distribution.[23]\ngene set analysis (GSA): GSA methods have been defined as a group of \"methods that aim to identify the pathways that are significantly impacted in a condition under study\"[6] or as \"tests which aim to detect pathways significantly enriched between two experimental conditions.\"[23] More specifically, GSA is an annotation-based approach that statistically compares experimental results to an annotated database in order to transform gene-level results into gene-set-level results. For example, a query gene set (a list of differentially expressed genes, or a rank of all gene\u2019s fold change) is mapped to a gene set reference database, using a particular statistical method, in order to explain the experimental results as a rank of significantly impacted pathways, functionally related gene sets, or ontology terms.\ngold standard: A \"perfect gold standard\" would be an error-free dataset that can be used as a synonym for truth (in our case, an omics dataset associated to a true ranking of pathways); however, in practice, we are limited to use \"imperfect\" or \"alloyed gold standards,\" which are datasets confidently linked to the truth, but not necessarily datasets lacking error.[40]\nnetwork interaction (NI): A subset of GSA methods that not only includes the given gene sets but also the gene products that interact with the members of such gene sets when located on top of an interaction or functional annotation network.\nover-representation analysis (ORA): A subset of GSA methods based on comparing a list of query genes (for example, up- or down-regulated genes) to a list of genes in a class or gene set, using a statistical test that detects over-representation. ORA \"statistically evaluates the fraction of genes in a particular pathway found among the set of genes showing changes in expression.\"[5]\npathway-topology-based (PT): A subset of GSA methods that weighs enrichment scores according to the position of a gene in a pathway. Only applies to pathway data and not to other types of gene sets.\nperformance: The value of a quantitative property (when compared to alternative methods or tools) that measures the agreement between the method's output and either empirical data, simulated data, or the output of another method.\npopularity: The frequency of use of a method or tool among members of a community.\nsimulation study: A systematic comparison of computational methods based on building artificial datasets that possess the properties that we specify for them.\n\nConstruction of the database \nThe Gene Set Analysis Reference Database (GSARefDB) was built from the following sources:\n1. Google and PubMed searches of keywords such as: \"pathway analysis,\" \"gene set analysis,\" and \"functional enrichment\" (approx. 10% of the records)\n2. Cross-references from all the collected papers and reviews (approx. 50% of the records)\n3. Email alerts received from NCBI and selected journals (approx. 40% of the records)\nThe information was classified into: (i) generic methods\/software\/platforms, usually dealing with mRNA datasets; (ii) reviews\/benchmark studies; (iii) genomic GSA, which includes GSA applied to enrichment of genomic regions (such as those coming from ChIP-seq, SNP, and methylation experiments); and (iv) ncRNA GSA, which includes methods dealing with miRNA and lncRNA datasets. All information in the database was manually extracted from the papers. Numbers of citations were extracted from Google Scholar. Only methods that associate omics data to annotated gene sets were included (see all types of included methods in the prior Fig. 1b). Bioinformatics methods that associate omics data to newly generated modules on a biological network were not included. GSARefDB was built both as an excel sheet and as a Shiny application (Figure 2a).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Screenshots of our tools for popularity and performance analysis of the GSA field. a GSARefDB: A screenshot of the R\/Shiny interface to GSARefDB, showing the options of searching by year, tool name, paper\u2019s first author, title, type of GSA, and programming language. b GSA BenchmarKING: One jupyter notebook containing an R workflow for benchmarking single-sample GSA methods, and one shiny app with the same purpose. Both tools display sensitivity, specificity, and precision plots for all the methods under study.\n\n\n\nDescriptive statistics \nPlots of summary statistics were generated using the \u201cggplot2\u201d R package. The R code is open and can be found here.\n\nPopularity rankings \nPopularity rankings in GSARefDB are constructed on a per paper basis, not per-method or per-tool. In order to build popularity rankings of methods or tools that are presented in multiple papers, we use the citation count of the most cited paper for that tool.\n\nPerformance study \nThe scientific validation approaches followed by the top 153 GSA tool papers in our database were manually reviewed (see Additional file 1: Tab 6). Validation, as performance, was defined as the success of a method on getting better scores than rival methods for a specific quantitative property that measures the agreement between the method's output and either empirical data, simulated data, or the output of another method. Our definition of validation does not include:\n(i) examples of application of the method followed by highlighting the reasonableness of the results (without comparing to other methods);\n(ii) arguments (usually statistical) stating that new assumptions are better than old ones, without any comparison to empirical or properly simulated data; or\n(iii) comparisons of capabilities between old and new software (such as implementing other algorithms or databases).\nDetailed procedures followed during the benchmark and simulation studies are explained in Additional file 2.\n\nConstruction of performance-measuring software \nThe \u201cGSA-BenchmarKING\u201d repository (see Figure 2b and the GSA BenchmarKING page) was created to store and share different tools to measure GSA method performance. For a benchmarking software to be accepted into the repository, it should: (i) be open software (ideally, a Jupyter Notebook, RStudio R Notebook, or Shiny app); (ii) have a clear reason for selecting the GSA methods included; for example, because all of them belong to the same type of methods; (iii) include both a gold standard dataset and options to upload user-selected gold standard datasets; (iv) include either a list of target pathways linked to the gold standard dataset or disease relevance scores per pathway for the diseases related to the gold standard; (v) give the user the option of selecting different benchmarking metrics (as a minimum, precision\/sensitivity, prioritization, and specificity\/FPR); (vi) give the possibility of selecting ensemble results; and (vii) give the possibility of adding new GSA methods to the code in the future.\nThe \u201cGSARefDB,\u201d \u201css-shiny,\u201d and \u201cgr-shiny\u201d apps were built using the \u201cShiny\u201d package. \u201cGSARefDB\u201d and \u201css-shiny\u201d were built using R 3.6.2, while \u201cgr-shiny\u201d was built using R 4.0.0. Open code can be accessed at GitHub here, here, and here.\n\nSupplementary information \n Additional file 1: GSA Reference DB\n Additional file 2: Detailed Performance Study\nAcknowledgements \nThe authors thank the support of both colleagues and students at Guangzhou Medical University and Guangzhou Institutes of Biomedicine and Health (Chinese Academy of Sciences). We also thank our two anonymous reviewers for their contributions.\n\nContributions \nCSX wrote the Shiny apps for GSARefDB and ss-shiny, and generated the figures. SJ programmed the Shiny app for gr-shiny. AM designed the project, built the reference database, and wrote the paper. All authors read and approved the final version of the paper.\n\nAvailability of data and materials \nAll datasets generated for this study are included in the supplementary material. Updated versions of the datasets will be available at the GSARefDB and GSA BenchmarKING pages.\n\nFunding \nThis work was funded by the Joint School of Life Sciences, Guangzhou Medical University and Guangzhou Institutes of Biomedicine and Health (Chinese Academy of Sciences). SJ is also funded by the China Postdoctoral Science Foundation Grant No. 2019M652847. The funders had no role in the design of the study, data collection, data analysis, interpretation of results, or writing of the paper.\n\nCompeting interests \nThe author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 Dixson, L.; Walter, H.; Schneider, M. et al. (2014). \"Retraction for Dixson et al., Identification of gene ontologies linked to prefrontal-hippocampal functional coupling in the human brain\". Proceedings of the National Academy of Sciences of the United States of America 111 (37): 13582. doi:10.1073\/pnas.1414905111. PMC PMC4169929. PMID 25197092. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4169929 .   \n\n\u2191 2.0 2.1 2.2 2.3 Mora, A. (2020). \"Gene set analysis methods for the functional interpretation of non-mRNA data-Genomic range and ncRNA data\". Briefings in Bioinformatics 21 (5): 1495-1508. doi:10.1093\/bib\/bbz090. PMID 31612220.   \n\n\u2191 3.0 3.1 Tavazoie, S.; Hughes, J.D.; Campbell, M.J. et al. (1999). \"Systematic determination of genetic network architecture\". Nature Genetics 22 (3): 281\u20135. doi:10.1038\/10343. PMID 10391217.   \n\n\u2191 4.0 4.1 4.2 Huang, D.W.; Sherman, B.T.; Lempicki, R.A. (2009). \"Bioinformatics enrichment tools: Paths toward the comprehensive functional analysis of large gene lists\". Nucleic Acids Research 37 (1): 1-13. doi:10.1093\/nar\/gkn923. PMC PMC2615629. PMID 19033363. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2615629 .   \n\n\u2191 5.0 5.1 5.2 5.3 5.4 Khatri, P.; Sirota, M.; Butte, A.J. (2012). \"Ten years of pathway analysis: Current approaches and outstanding challenges\". PLoS Computational Biology 8 (2): e1002375. doi:10.1371\/journal.pcbi.1002375. PMC PMC3285573. PMID 22383865. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3285573 .   \n\n\u2191 6.0 6.1 Mitrea, C.; Taghavi, Z.; Bokanizad, B. et al. (2013). \"Methods and approaches in the topology-based analysis of biological pathways\". Frontiers in Physiology 4: 278. doi:10.3389\/fphys.2013.00278. PMC PMC3794382. PMID 24133454. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3794382 .   \n\n\u2191 Mora, A. (26 March 2021). \"GSARefDB\". GSA Central. https:\/\/gsa-central.github.io\/gsarefdb.html .   \n\n\u2191 8.0 8.1 Huang, D.W.; Sherman, B.T.; Tan, Q. et al. (2007). \"DAVID Bioinformatics Resources: Expanded annotation database and novel algorithms to better extract biology from large gene lists\". Nucleic Acids Research 35 (Web Server Issue): W169-75. doi:10.1093\/nar\/gkm415. PMC PMC1933169. PMID 17576678. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1933169 .   \n\n\u2191 Subramanian, A.; Tamayo, P.; Mootha, V.K. et al. (2005). \"Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles\". Proceedings of the National Academy of Sciences of the United States of America 102 (43): 15545-50. doi:10.1073\/pnas.0506580102. PMC PMC1239896. PMID 16199517. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1239896 .   \n\n\u2191 Draghici, S.; Khatri, P.; Laurentiu, A. et al. (2007). \"A systems biology approach for pathway level analysis\". Genome Research 17 (10): 1537\u201345. doi:10.1101\/gr.6202607. PMC PMC1987343. PMID 17785539. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1987343 .   \n\n\u2191 \u0150sz, \u00c1.; Pongor, L.S.; Szirmai, D. et al. (2019). \"A snapshot of 3649 Web-based services published between 1994 and 2017 shows a decrease in availability after two years\". Briefings in Bioinformatics 20 (3): 1004-1010. doi:10.1093\/bib\/bbx159. PMC PMC6585384. PMID 29228189. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6585384 .   \n\n\u2191 12.0 12.1 Yu, G.; Wang,L.-G.; Hanf, Y. et al. (2012). \"clusterProfiler: an R package for comparing biological themes among gene clusters\". OMICS 16 (5): 284\u20137. doi:10.1089\/omi.2011.0118. PMC PMC3339379. PMID 22455463. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3339379 .   \n\n\u2191 Kuleshov, M.V.; Jones, M.R.; Rouillard, A.D. et al. (2016). \"Enrichr: A comprehensive gene set enrichment analysis web server 2016 update\". Nucleic Acids Research 44 (W1): W90\u20147. doi:10.1093\/nar\/gkw377. PMC PMC4987924. PMID 27141961. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987924 .   \n\n\u2191 Young, M.D.; Wakefield, M.J.; Smyth, G.K. et al. (2010). \"Gene ontology analysis for RNA-seq: accounting for selection bias\". Genome Biology 11 (2): R14. doi:10.1186\/gb-2010-11-2-r14. PMC PMC2872874. PMID 20132535. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2872874 .   \n\n\u2191 Bindea, G.; Mlecnik, B.; Hackl, H. et al. (2009). \"ClueGO: A Cytoscape plug-in to decipher functionally grouped gene ontology and pathway annotation networks\". Bioinformatics 25 (8): 1091\u20133. doi:10.1093\/bioinformatics\/btp101. PMC PMC2666812. PMID 19237447. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2666812 .   \n\n\u2191 Giannoulatouet, E.; Park, S.-H.; Humphreys, D.T. al. (2014). \"Verification and validation of bioinformatics software without a gold standard: A case study of BWA and Bowtie\". BMC Bioinformatics 15 (Suppl. 16): S15. doi:10.1186\/1471-2105-15-S16-S15. PMC PMC4290646. PMID 25521810. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4290646 .   \n\n\u2191 Curtis, R.K.; Oresic, M.; Vidal-Puig, A. (2005). \"Pathways to the analysis of microarray data\". Trends in Biotechnology 23 (8): 429-35. doi:10.1016\/j.tibtech.2005.05.011. PMID 15950303.   \n\n\u2191 Tarca, A.L.; Draghici, S.; Bhatti, G. et al. (2012). \"Down-weighting overlapping genes improves gene set analysis\". BMC Bioinformatics 13: 136. doi:10.1186\/1471-2105-13-136. PMC PMC3443069. PMID 22713124. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3443069 .   \n\n\u2191 Tomfohr, J.; Lu, J.; Kepler, T.B. (2005). \"Pathway level analysis of gene expression using singular value decomposition\". BMC Bioinformatics 6: 225. doi:10.1186\/1471-2105-6-225. PMC PMC1261155. PMID 16156896. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1261155 .   \n\n\u2191 Tarca, A.L.; Draghici, S.; Khatri, P. et al. (2009). \"A novel signaling pathway impact analysis\". Bioinformatics 25 (1): 75\u201382. doi:10.1093\/bioinformatics\/btn577. PMC PMC2732297. PMID 18990722. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2732297 .   \n\n\u2191 21.0 21.1 21.2 Naeem, H.; Zimmer, R.; Tavakkolkhah, P. et al. (2012). \"Rigorous assessment of gene set enrichment tests\". Bioinformatics 28 (11): 1480-6. doi:10.1093\/bioinformatics\/bts164. PMID 22492315.   \n\n\u2191 22.0 22.1 22.2 22.3 22.4 22.5 Tarca, A.L.; Bhatti, G.; Romero, R. (2013). \"A comparison of gene set analysis methods in terms of sensitivity, prioritization and specificity\". PLoS One 8 (11): e79217. doi:10.1371\/journal.pone.0079217. PMC PMC3829842. PMID 24260172. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829842 .   \n\n\u2191 23.0 23.1 23.2 23.3 23.4 23.5 Bayerlov\u00e1, M.; Jung, K.; Kramer, F. et al. (2015). \"Comparative study on gene set and pathway topology-based enrichment methods\". BMC Bioinformatics 16: 334. doi:10.1186\/s12859-015-0751-5. PMC PMC4618947. PMID 26489510. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4618947 .   \n\n\u2191 Jaakkola, M.K.; Elo, L.L. (2016). \"Empirical comparison of structure-based pathway methods\". Briefings in Bioinformatics 17 (2): 336\u201345. doi:10.1093\/bib\/bbv049. PMC PMC4793894. PMID 26197809. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4793894 .   \n\n\u2191 25.0 25.1 De Meyer, S. (2016). \"Assessing the performance of network crosstalk analysis combined with clustering\" (PDF). Univeriteit Gent. https:\/\/libstore.ugent.be\/fulltxt\/RUG01\/002\/272\/484\/RUG01-002272484_2016_0001_AC.pdf .   \n\n\u2191 Lim, S.; Lee, S.; Jung, I. et al. (2020). \"Comprehensive and critical evaluation of individualized pathway activity measurement tools on pan-cancer data\". Briefings in Bioinformatics 21 (1): 36\u201346. doi:10.1093\/bib\/bby125. PMID 30462155.   \n\n\u2191 27.0 27.1 Nguyen, T.-M.; Shafi, A.; Nguyen, T. et al. (2019). \"Identifying significantly impacted pathways: A comprehensive review and assessment\". Genome Biology 20 (1): 203. doi:10.1186\/s13059-019-1790-4. PMC PMC6784345. PMID 31597578. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6784345 .   \n\n\u2191 Ma, J.; Shojaie, A.; Michailidis, G. (2019). \"A comparative study of topology-based pathway enrichment analysis methods\". BMC Bioinformatics 20 (1): 546. doi:10.1186\/s12859-019-3146-1. PMC PMC6829999. PMID 31684881. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6829999 .   \n\n\u2191 29.0 29.1 29.2 Zyla, J.; Marczyk, M.; Doaszewska, T. et al. (2019). \"Gene set enrichment for reproducible science: Comparison of CERNO and eight other algorithms\". Bioinformatics 35 (24): 5146\u201354. doi:10.1093\/bioinformatics\/btz447. PMC PMC6954644. PMID 31165139. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6954644 .   \n\n\u2191 30.0 30.1 Geistlinger, L.; Csaba, G.; Santarelli, M. et al. (2021). \"Toward a gold standard for benchmarking gene set enrichment analysis\". Briefings in Bioinformatics 22 (1): 545\u201356. doi:10.1093\/bib\/bbz158. PMC PMC7820859. PMID 32026945. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7820859 .   \n\n\u2191 Glaab, E.; Baudot, A.; Krasnogor, N. et al. (2012). \"EnrichNet: Network-based gene set enrichment analysis\". Briefings in Bioinformatics 28 (18): i451-i457. doi:10.1093\/bioinformatics\/bts389. PMC PMC3436816. PMID 22962466. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3436816 .   \n\n\u2191 Dong, X.; Hao, Y.; Wang, X. et al. (2016). \"LEGO: A novel method for gene set over-representation analysis by incorporating network-based gene weights\". Scientific Reports 6: 18871. doi:10.1038\/srep18871. PMC PMC4707541. PMID 26750448. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4707541 .   \n\n\u2191 Alhamdoosh, M.; Law, C.W.; Tian, L. et al. (2017). \"Easy and efficient ensemble gene set testing with EGSEA\". F1000Research 6: 2010. doi:10.12688\/f1000research.12544.1. PMC PMC5747338. PMID 29333246. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5747338 .   \n\n\u2191 Kokemuller, N. (28 January 2019). \"Why Do People Buy Brand Names?\". Chron. https:\/\/smallbusiness.chron.com\/people-buy-brand-names-69654.html .   \n\n\u2191 Barab\u00e1si, A.-L. (2018). The Formula: The Universal Laws of Success. Little, Brown and Company. ISBN 9780316505499.   \n\n\u2191 Lee, J.; Jo, K.; Lee, S. et al. (2016). \"Prioritizing biological pathways by recognizing context in time-series gene expression data\". BMC Bioinformatics 17 (Suppl 17): 477. doi:10.1186\/s12859-016-1335-8. PMC PMC5259824. PMID 28155707. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5259824 .   \n\n\u2191 Vitek, O.; Robinson, M.D., ed. (2020). \"Benchmarking Studies\". Genome Biology. https:\/\/www.biomedcentral.com\/collections\/benchmarkingstudies .   \n\n\u2191 \"PLOS Computational Biology Benchmarking Collection\". PLoS. 29 November 2018. https:\/\/collections.plos.org\/collection\/benchmarking\/ .   \n\n\u2191 Efroni, S.; Schaefer, C.F.; Buetow, K.H. (2007). \"Identification of key processes underlying cancer phenotypes using biologic pathway analysis\". PLoS One 2 (5): e425. doi:10.1371\/journal.pone.0000425. PMC PMC1855990. PMID 17487280. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1855990 .   \n\n\u2191 Wacholder, S.; Armstrong, B.; Hartge, P. (1993). \"Validation studies using an alloyed gold standard\". American Journal of Epidemiology 137 (11): 1251\u20138. doi:10.1093\/oxfordjournals.aje.a116627. PMID 8322765.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\">https:\/\/www.limswiki.org\/index.php\/Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 21 April 2021, at 23:37.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 437 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","7ef870605af2fae6ca37ea1b35850e50_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Popularity_and_performance_of_bioinformatics_software_The_case_of_gene_set_analysis skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Popularity and performance of bioinformatics software: The case of gene set analysis<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: Gene set analysis (GSA) is arguably the method of choice for the functional interpretation of omics results. This work explores the popularity and the performance of all the GSA methodologies and software published during the 20 years since its inception. \"Popularity\" is estimated according to each paper's citation counts, while \"performance\" is based on a comprehensive evaluation of the validation strategies used by papers in the field, as well as the consolidated results from the existing benchmark studies.\n<\/p><p><b>Results<\/b>: Regarding popularity, data is collected into an online open database (\"GSARefDB\") which allows browsing bibliographic and method-descriptive <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> from 503 GSA paper references; regarding performance, we introduce a repository of <a href=\"https:\/\/www.limswiki.org\/index.php\/Jupyter_Notebook\" title=\"Jupyter Notebook\" class=\"wiki-link\" data-key=\"26fd35430c10e009a142bbab5dbf617a\">Jupyter Notebook<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflows<\/a> and Shiny apps for automated benchmarking of GSA methods (\u201cGSA-BenchmarKING\u201d). After comparing popularity versus performance, results show discrepancies between the most popular and the best performing GSA methods.\n<\/p><p><b>Conclusions<\/b>: The above-mentioned results call our attention towards the nature of the tool selection procedures followed by researchers and raise doubts regarding the quality of the functional interpretation of biological datasets in current biomedical studies. Suggestions for the future of the functional interpretation field are made, including strategies for education and discussion of GSA tools, better validation and benchmarking practices, reproducibility, and functional re-analysis of previously reported data.\n<\/p><p><b>Keywords<\/b>: bioinformatics, pathway analysis, gene set analysis, benchmark, GSEA\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">Bioinformatics<\/a> method and software selection is an important problem in biomedical research, due to the possible consequences of choosing the wrong methods among the existing myriad of computational methods and software available. Errors in software selection may include the use of outdated or suboptimal methods (or reference databases) or misunderstanding the parameters and assumptions behind the chosen methods. Such errors may affect the conclusions of the entire research project and nullify the efforts made on the rest of the experimental and computational pipeline.<sup id=\"rdp-ebb-cite_ref-DixsonRetract14_1-0\" class=\"reference\"><a href=\"#cite_note-DixsonRetract14-1\">[1]<\/a><\/sup>\n<\/p><p>This paper discusses two main factors that motivate researchers to make method or software choices, that is, the popularity (defined as the perceived frequency of use of a tool among members of the community) and the performance (defined as a quantitative quality indicator measured and compared to alternative tools). This study is focused on the field of \u201cgene set analysis\u201d (GSA), where the popularity and performance of bioinformatics software show discrepancies, and therefore the question appears whether biomedical sciences have been using the best available GSA methods or not.\n<\/p><p>GSA is arguably the most common procedure for functional interpretation of omics data, and, for the purposes of this paper, we define it as the comparison of a query gene set (a list or a rank of differentially expressed genes, for example) to a reference database, using a particular statistical method, in order to interpret it as a rank of significant pathways, functionally related gene sets, or <a href=\"https:\/\/www.limswiki.org\/index.php\/Ontology_(information_science)\" title=\"Ontology (information science)\" class=\"wiki-link\" data-key=\"52d0664bde4b458e81fbc128b911a4a6\">ontology<\/a> terms. Such definition includes the categories that have been traditionally called \"gene set analysis,\" \"pathway analysis,\" \"ontology analysis,\" and \"enrichment analysis.\" All GSA methods have a common goal, which is the interpretation of biomolecular data in terms of annotated gene sets, while they differ depending on characteristics of the computational approach (for more details, see the \"Methods\" section, as well as Fig. 1 of Mora<sup id=\"rdp-ebb-cite_ref-MoraGene20_2-0\" class=\"reference\"><a href=\"#cite_note-MoraGene20-2\">[2]<\/a><\/sup>). \n<\/p><p>GSA has arrived to 20 years of existence since the original paper of Tavazoie <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TavazoieSystem99_3-0\" class=\"reference\"><a href=\"#cite_note-TavazoieSystem99-3\">[3]<\/a><\/sup>, and many statistical methods and software tools have been developed during this time. A popular review paper listed 68 GSA tools<sup id=\"rdp-ebb-cite_ref-HuangBioinfo09_4-0\" class=\"reference\"><a href=\"#cite_note-HuangBioinfo09-4\">[4]<\/a><\/sup>, while a second review reported an additional 33 tools<sup id=\"rdp-ebb-cite_ref-KhatriTenYears12_5-0\" class=\"reference\"><a href=\"#cite_note-KhatriTenYears12-5\">[5]<\/a><\/sup>, and a third paper 22 tools.<sup id=\"rdp-ebb-cite_ref-MitreaMeth13_6-0\" class=\"reference\"><a href=\"#cite_note-MitreaMeth13-6\">[6]<\/a><\/sup> We have built the most comprehensive list of references to date (503 papers), and we have quantified each paper\u2019s influence according to their current number of citations (see Additional file 1 and Mora's GSARefDB<sup id=\"rdp-ebb-cite_ref-GSARedDB_7-0\" class=\"reference\"><a href=\"#cite_note-GSARedDB-7\">[7]<\/a><\/sup>). The most common GSA methods include Over-Representation Analysis (ORA), such as that found with DAVID<sup id=\"rdp-ebb-cite_ref-HuangDAVID07_8-0\" class=\"reference\"><a href=\"#cite_note-HuangDAVID07-8\">[8]<\/a><\/sup>; Functional Class Scoring (FCS), such as that found with GSEA<sup id=\"rdp-ebb-cite_ref-SubramanianGene05_9-0\" class=\"reference\"><a href=\"#cite_note-SubramanianGene05-9\">[9]<\/a><\/sup>; and Pathway-Topology-based (PT) methods, such as that found with SPIA.<sup id=\"rdp-ebb-cite_ref-DraghiciASys07_10-0\" class=\"reference\"><a href=\"#cite_note-DraghiciASys07-10\">[10]<\/a><\/sup> All been extensively reviewed. In order to know more about them, the reader may consult any of the 62 published reviews documented in Additional file 1. We have also recently reviewed other types of GSA methods.<sup id=\"rdp-ebb-cite_ref-MoraGene20_2-1\" class=\"reference\"><a href=\"#cite_note-MoraGene20-2\">[2]<\/a><\/sup>\n<\/p><p>The first part of the analysis is a study on GSA method and software popularity based on a comprehensive database of 503 papers for all methods, tools, platforms, reviews, and benchmarks of the GSA field, collectively cited 134,222 times between 1999 and 2019, including their popularity indicators and other relevant characteristics. The second part is a study on performance based on the validation procedures reported in the papers, introducing new methods together with all the existing independent benchmark studies in the above-mentioned database. Instead of recommending one single GSA method, we focus on discussing better benchmarking practices and generating benchmarking tools that follow such practices. Together, both parts of the study allow us to compare the popularity and performance of GSA tools but also to explore possible explanations for the popularity phenomenon and the problems that limit the execution and adoption of independent performance studies. In the end, some practices are suggested to guarantee that bioinformatics software selection is guided by the most appropriate metrics.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Popularity\">Popularity<\/span><\/h3>\n<p>The number of paper citations has been used as a simple (yet imperfect) measure of a GSA method\u2019s popularity. We collected 350 references to GSA papers of methods, software or platforms (Additional file 1: Tab 1), as well as 91 references to GSA papers for non-mRNA omics tools (Additional file 1: Tabs 3, 4), and 62 GSA reviews or benchmark studies (Additional file 1: Tabs 2, 3, 4), which have been organized into a manually curated open database (GSARefDB). GSARefDB can be either downloaded or accessed through a <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/gsarefdb.html\" target=\"_blank\">Shiny interface<\/a>. Figure 1 summarizes some relevant information from the database (GSARefDB v.1.0). \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Xie_BMCBioinfo21_22.png\" class=\"image wiki-link\" data-key=\"c631c19e479269fed7cef7599058ef8b\"><img alt=\"Fig1 Xie BMCBioinfo21 22.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/94\/Fig1_Xie_BMCBioinfo21_22.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Statistics from GSARefDB v.1.0 (for more recent statistics, visit our <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/gsarefdb.html\" target=\"_blank\">website<\/a>). <b>a<\/b> Number of GSA publications per year. <b>b<\/b> Number of publications per type of GSA method. <b>c<\/b> Number of publications per used programming language. <b>d<\/b> Number of citations per used programming language. <b>e<\/b> Website availability. <b>f<\/b> Number of publications per reported validation method.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The citation count shows that the most influential GSA method in history is Gene Set Enrichment Analysis (GSEA), published in 2003, with more than twice as many citations as its follower, the ORA platform called DAVID (17,877 versus 7,500 citations). In general, the database shows that the field contains a few extremely popular papers and many papers with low popularity. Figure 1b shows that the list of tools is mainly composed of ORA and FCS methods, while the newer and lesser known PT and Network Interaction (NI) methods are less common (and generally found at the bottom of the popularity rank).\n<\/p><p>It could be hypothesized that the popularity of a GSA tool does not always depend on being the best for that particular project, and it could be related to variables such as its user-friendliness instead. The database allows us to compute citations-per-programming-language, which we use as an approximation to friendliness. Figure 1c shows that the majority of the GSA papers correspond to <a href=\"https:\/\/www.limswiki.org\/index.php\/R_(programming_language)\" title=\"R (programming language)\" class=\"wiki-link\" data-key=\"1b0aa598f071aca4c5b4ee08d8bb2bde\">R tools<\/a>, but, in spite of that, Fig. 1d shows that most of the citations correspond to web platforms followed by stand-alone applications, which are friendlier to users. Worth mentioning, the last column of the database shows that there are a large number of tools that are not maintained anymore, with broken web links to tools or databases, which makes their evaluation impossible. This phenomenon is a common bioinformatics problem that has also been reported elsewhere.<sup id=\"rdp-ebb-cite_ref-.C5.90szASnap19_11-0\" class=\"reference\"><a href=\"#cite_note-.C5.90szASnap19-11\">[11]<\/a><\/sup> Figure 1e shows that one-third of the reported links in GSA papers are now broken links.\n<\/p><p>Besides ranking papers according to their all-time popularity, a current-popularity rank was also built (Additional file 1: Tab 5). To accomplish that, a version of the database generated in May 2018 was compared to a version of the database generated in April 2019. The \"current popularity\" rank revealed the same trends as the \"overall\" rank. GSEA is still, by far, the most popular method. The other tools that are being currently cited are clusterProfiler<sup id=\"rdp-ebb-cite_ref-YuCluster12_12-0\" class=\"reference\"><a href=\"#cite_note-YuCluster12-12\">[12]<\/a><\/sup>, Enrichr<sup id=\"rdp-ebb-cite_ref-KuleshovEnrichr16_13-0\" class=\"reference\"><a href=\"#cite_note-KuleshovEnrichr16-13\">[13]<\/a><\/sup>, GOseq<sup id=\"rdp-ebb-cite_ref-YoungGene10_14-0\" class=\"reference\"><a href=\"#cite_note-YoungGene10-14\">[14]<\/a><\/sup>, DAVID<sup id=\"rdp-ebb-cite_ref-HuangDAVID07_8-1\" class=\"reference\"><a href=\"#cite_note-HuangDAVID07-8\">[8]<\/a><\/sup>, and ClueGO<sup id=\"rdp-ebb-cite_ref-BindeaClue09_15-0\" class=\"reference\"><a href=\"#cite_note-BindeaClue09-15\">[15]<\/a><\/sup>, followed by GOrilla, KOBAS, BiNGO, ToppGene, GSVA, WEGO, agriGO, and WebGestalt. ORA and FCS methods are still the most popular ones, with 3534 combined citations for all ORA methods and 2185 citations for all FCS methods, while PT and NI methods have 111 and 50 combined citations respectively. In contrast, single-sample methods have 278 combined citations, while time-course methods have 67. Regarding reviews, an extremely popular paper from 2009<sup id=\"rdp-ebb-cite_ref-HuangBioinfo09_4-1\" class=\"reference\"><a href=\"#cite_note-HuangBioinfo09-4\">[4]<\/a><\/sup> is still currently the most popular, even though it doesn\u2019t take into account the achievements of the last 10 years.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Performance\">Performance<\/span><\/h3>\n<p>The subject of <a href=\"https:\/\/www.limswiki.org\/index.php\/Software_verification_and_validation\" title=\"Software verification and validation\" class=\"wiki-link\" data-key=\"14bdd953b2bde31c520ba57b1002195e\">validation<\/a> of bioinformatics software deserves more attention.<sup id=\"rdp-ebb-cite_ref-GiannoulatouVerif14_16-0\" class=\"reference\"><a href=\"#cite_note-GiannoulatouVerif14-16\">[16]<\/a><\/sup> A review of the scientific validation approaches followed by the top 153 GSA tool papers in the database (Additional file 1: Tab 6) found multiple validation strategies that were classified into 19 categories. 61 out of the 153 papers include a validation procedure, and the most commonly found validation strategy is \u201cConsistency with biological knowledge,\u201d defined as the fact that our method\u2019s results explain the knowledge in the field better than the rival methods (which is commonly accomplished through a literature search). Other common strategies (though less common) are the comparisons of the number of hits, classification accuracy, and consistency of results between similar samples. Important strategies, such as comparing statistical power, benchmark studies, and simulations, are less used. The least used strategies include experimental confirmation of predictions and semi-blind procedures where a person collects samples and another person applies the tool to guess tissue or condition. Our results have been summarized in Fig. 1f (above) and Additional file 1: Tab 6. We can see that the frequency of use of the above-mentioned validation strategies is inversely proportional to their reliability. For example, commonly used strategies such as \u201cconsistency with biological knowledge\u201d can be subjective, and comparing the number of hits of our method with other methods on a Venn diagram<sup id=\"rdp-ebb-cite_ref-CurtisPath05_17-0\" class=\"reference\"><a href=\"#cite_note-CurtisPath05-17\">[17]<\/a><\/sup> is a measure of agreement, not truth. On the other hand, the least used strategies, such as experimental confirmation or benchmark and simulation studies, are the better alternatives.\n<\/p><p>The next step of our performance study was a review of all the independent benchmark and simulation studies existing in the GSA field, whose references are collected in Additional file 1: Tab 2. Table 1 summarizes 10 benchmark studies of GSA methods, with different sizes, scopes, and method recommendations. A detailed description and discussion of each of the benchmarking studies can be found in Additional file 2. The sizes of all benchmarking studies are small when compared to the amount of existing methods that we have mentioned before, while their lists of best performing methods show little overlap. Only a few methods are mentioned as best performers in more than one study, including ORA methods (such as hypergeometric)<sup id=\"rdp-ebb-cite_ref-TavazoieSystem99_3-1\" class=\"reference\"><a href=\"#cite_note-TavazoieSystem99-3\">[3]<\/a><\/sup>, FCS methods (such as PADOG)<sup id=\"rdp-ebb-cite_ref-TarcaDown12_18-0\" class=\"reference\"><a href=\"#cite_note-TarcaDown12-18\">[18]<\/a><\/sup>, SS methods (such as PLAGE)<sup id=\"rdp-ebb-cite_ref-TomfohrPath05_19-0\" class=\"reference\"><a href=\"#cite_note-TomfohrPath05-19\">[19]<\/a><\/sup>, and PT methods (such as SPIA\/ROntoTools).<sup id=\"rdp-ebb-cite_ref-TarcaANovel09_20-0\" class=\"reference\"><a href=\"#cite_note-TarcaANovel09-20\">[20]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 1.<\/b> Ten benchmark studies of GSA methods, from 2012 to 2020, showing a plurality of scopes, sizes, and method recommendations. Details on each study can be found in Additional file 2. ORA, over-representation analysis; FCS, functional class scoring; PT, pathway topology-based; SS, single-sample; NI, network interaction.\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Study reference\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Scope\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Size\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Criteria\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Best performing methods\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Naeem <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-NaeemRigor12_21-0\" class=\"reference\"><a href=\"#cite_note-NaeemRigor12-21\">[21]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ORA and FCS methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Method\u2019s AUC (evaluated by predicting targets of TFs and miRNAs)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ANOVA, Z-SCORE, and Wilcoxon\u2019s rank sum (WRS)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tarca <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TarcaAComp13_22-0\" class=\"reference\"><a href=\"#cite_note-TarcaAComp13-22\">[22]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ORA, FCS, and SS methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Prioritization, Sensitivity, and FPR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GLOBALTEST and PLAGE (sensitivity), PADOG and ORA (prioritization), and CAMERA (FPR). Author\u2019s general recommendation: PLAGE, GLOBALTEST, and PADOG.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bayerlov\u00e1 <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Bayerlov.C3.A1Comp15_23-0\" class=\"reference\"><a href=\"#cite_note-Bayerlov.C3.A1Comp15-23\">[23]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ORA, FCS, and PT methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sensitivity and prioritization (for benchmark), and Sensitivity, specificity, and accuracy (for simulations of pathway overlap)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">For benchmark: CePaGSA (sensitivity) and PathNet (prioritization). For simulation -original pathways: CePAGSA (sensitivity), WRS (specificity), and WRS (accuracy). For simulation -non-overlapping pathways: KS (sensitivity), and SPIA, CePaORA, CePaGSA, and PathNet (specificity and accuracy).\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Jaakkola <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-JaakkolaEmp16_24-0\" class=\"reference\"><a href=\"#cite_note-JaakkolaEmp16-24\">[24]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ORA, FCS, and PT methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Consistency of significant pathways between datasets, and Sensitivity\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SPIA and CePaORA (consistency), and SPIA, CePaORA, and NetGSA (sensitivity). Author\u2019s general recommendation: SPIA.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">De Meyer<sup id=\"rdp-ebb-cite_ref-DeMeyerAssess16_25-0\" class=\"reference\"><a href=\"#cite_note-DeMeyerAssess16-25\">[25]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ORA, FCS, and NI methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Prioritization, Sensitivity, and Specificity\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PADOG (specificity) and BinoX (sensitivity).\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lim <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-LimComp20_26-0\" class=\"reference\"><a href=\"#cite_note-LimComp20-26\">[26]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SS\/Pathway-activity methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Classification performance, preservation of data structure, robustness to noise, and reproducibility between pathway databases\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ESEA, Pathifier, SAS, and PADOG (classification tasks), Pathifier and PLAGE (data structure), ssGSEA (robustness), and individPath, Pathifier, and SAS (reproducibility). Author\u2019s general recommendation: Pathifier, SAS, and individPath.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nguyen <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-NguyenIndent19_27-0\" class=\"reference\"><a href=\"#cite_note-NguyenIndent19-27\">[27]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ORA, FCS, and PT methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">In order of importance: Number of biased pathways, Prioritization, Method\u2019s AUC, and sensitivity (evaluated using both disease target pathways and KO data)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GSEA (bias), PADOG (prioritization), ROntoTools (AUC), and CePaGSA (p-values). Author\u2019s general recommendation: ROntoTools.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ma <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MaAComp19_28-0\" class=\"reference\"><a href=\"#cite_note-MaAComp19-28\">[28]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">FCS, PT, and NI methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ranking of empirical powers\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DEGraph, followed by PathNet and NetGSA\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Zyla <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ZylaGene19_29-0\" class=\"reference\"><a href=\"#cite_note-ZylaGene19-29\">[29]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ORA, FCS, and SS methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sensitivity, FPR, prioritization, computational time, and reproducibility\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PLAGE (sensitivity), ORA and PADOG (specificity\/FPR), PADOG (prioritization), and CERNO (reproducibility).\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Geistlinger <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Geistlinger_30-0\" class=\"reference\"><a href=\"#cite_note-Geistlinger-30\">[30]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ORA, FCS, and SS methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10 methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sensitivity, computational time, and phenotype relevance score\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Author\u2019s general recommendation: ROAST and GSVA (for self-contained hypothesis). ORA and PADOG (for competitive hypothesis).\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>It is crucial to notice that the previous studies just covered a small fraction of the entire universe of GSA methods. Also, that there is little overlap between the sets of methods included in different studies, but nevertheless we can still find inconsistencies between results, such as the GSVA, Pathifier, and hypergeometric methods, which are reported both as best performer and as poor performer in different benchmarks. From all the above-mentioned high-performance tools, only \u201cORA\u201d appears in the top 20 currently most popular tools (Additional file 1: Tab 5), suggesting that there is a divorce between popularity and performance (see also Additional file 2: Table 2).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Popularity_versus_performance\">Popularity versus performance<\/span><\/h3>\n<p>In general terms, it is evident that performance studies are still few, small, inconsistent, and dependent on the quality of the benchmarks; however, they tend to recommend tools different to the popular and friendly ones.\n<\/p><p>GSEA is one of the most important landmarks in GSA history and the most striking example of the contradictions between popularity and performance metrics of GSA tools. With one exception, none of the recent benchmarks under analysis has reported GSEA to be the best performing method; however, GSEA is still both the all-time and the currently most used tool. Besides that, numerous methods (most of them not included in the previous benchmarks) report that they specifically outperform GSEA in at least one of many possible ways; such methods are highlighted on Additional file 1: Tab 7 with the tag \"COMPARED\". In addition, there have been plenty of developments to the GSEA method itself; for example, alternative functions to score the gene set, alternative options to the permutation step to finding <i>p<\/i>-values, or using GSEA as part of an extended method (see methods at Additional file 1: Tab 7, with the tag \"PART OF METHOD\"), which reportedly outperform the original GSEA. We have identified a total of 79 papers in those categories (Additional file 1: Tab 7). However, in spite of that, GSEA's overwhelming popularity compared to any other method would suggest otherwise. We can also verify that, with the exception of ORA, most of the above-mentioned high-performing methods tend to occupy lower places in the current popularity ranking (Additional file 2: Table 2).\n<\/p><p>It is hard to estimate how many conclusions from how many articles should be rectified if we rigorously apply our current knowledge on GSA and use the best performing methods in every paper that ever used GSA. However, taking into account that our database (Additional file 1) registers 503 GSA software and tool papers that have been collectively cited 134,222 times, there is a valid concern regarding the need to pay more attention to both the method performance studies and the performance-based tool selection in GSA. Otherwise, the last step of the omics data analysis process may undermine the efforts on the rest of the pipeline.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Performance_tools\">Performance tools<\/span><\/h3>\n<p>The previous observation regarding the difference between the most popular and the best performing GSA methods could be incorrect, given that existing benchmarks have followed different methodologies and the proper way to benchmark GSA methods is a discussion in itself. Therefore, we have also reviewed and discussed the existing benchmarking and simulation strategies, in order to extract a few ideas regarding good benchmark practices.\n<\/p><p>Rigorous benchmark studies are not a straightforward task, given that there is no accepted gold standard to be used in a comparison of GSA tools.<sup id=\"rdp-ebb-cite_ref-DeMeyerAssess16_25-1\" class=\"reference\"><a href=\"#cite_note-DeMeyerAssess16-25\">[25]<\/a><\/sup> Reviewed here are three strategies to build such a gold standard. The first strategy is to apply several different enrichment tools to the same gene sets and use the intersection of the results as a gold-standard. One example of this is EnrichNet<sup id=\"rdp-ebb-cite_ref-GlaabEnrich12_31-0\" class=\"reference\"><a href=\"#cite_note-GlaabEnrich12-31\">[31]<\/a><\/sup>, which obtains a \u201cset of high confidence benchmark pathways\u201d as the intersection between the top 100 ranks generated by the SAM-GS and GAGE methods. Such a procedure is questionable, as the chosen methods are far from being considered the best (as seen before), and the procedure is more related to agreement than truth. The second strategy is the use of disease datasets clearly associated with one pathway as a gold-standard. For example, Tarca <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TarcaAComp13_22-1\" class=\"reference\"><a href=\"#cite_note-TarcaAComp13-22\">[22]<\/a><\/sup> compiled 42 microarray datasets with both healthy and disease-associated samples, where the disease was associated with a KEGG pathway (and, therefore, such \u201ctarget pathway\u201d should be found significant).<sup id=\"rdp-ebb-cite_ref-TarcaAComp13_22-2\" class=\"reference\"><a href=\"#cite_note-TarcaAComp13-22\">[22]<\/a><\/sup> The third strategy is to leverage knowledge on gene regulation effects. Some authors have used gene sets built from the known targets of specific transcription factors and miRNAs, and then try to predict the changes after over-expression or deletion of such regulators<sup id=\"rdp-ebb-cite_ref-NaeemRigor12_21-1\" class=\"reference\"><a href=\"#cite_note-NaeemRigor12-21\">[21]<\/a><\/sup>, while some others have used datasets with mouse KOs: Pathways containing the KO gene were considered as target pathways, while the others were considered as negatives.<sup id=\"rdp-ebb-cite_ref-NguyenIndent19_27-1\" class=\"reference\"><a href=\"#cite_note-NguyenIndent19-27\">[27]<\/a><\/sup> Geistlinger <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Geistlinger_30-1\" class=\"reference\"><a href=\"#cite_note-Geistlinger-30\">[30]<\/a><\/sup> have recently introduced a modification of such approach in which they not only look at \u201ctarget pathways\u201d matching the original diseases, but instead create a \u201cgene set relevance ranking\u201d for each disease. To build such rankings, the authors used MalaCards gene scores for disease relevance, which are based on both experimental and bibliometric evidence; then, they used the gene scores to build combined gene set relevance scores for all GO and KEGG terms (using the GeneAnalytics tool). As a result, instead of benchmarking against some \u201ctarget pathways\u201d, the benchmark is done against a \u201cpathway relevance ranking\u201d per disease. Table 2 assesses different performance measurement approaches according to their objectivity, reproducibility, and scalability.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 2.<\/b> Objectivity, Reproducibility, and Scalability of the main performance criteria. Objectivity refers to the results not depending on human interpretation. Reproducibility refers to any researcher being able to find the same results by following the same procedures. Scalability refers to the possibility of easily applying the procedure to an increasingly higher number of methods and datasets.\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Criteria\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Objectivity\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Reproducibility\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Scalability\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Drawbacks\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tool agreement\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Low\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Low\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Subjective\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Consistency between similar samples\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Low\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Subjective\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Consistency with biological knowledge (Literature search)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Low\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Low\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Low\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Subjective\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Benchmark (target pathways as gold standard)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Centered on true positives\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Benchmark (pathway relevance ranking as gold standard)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Benchmark (KO\/perturbation data as gold standard)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Simulations\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Low\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Low\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Human-designed datasets may be unrealistic\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>A second issue is the selection of the benchmark metrics that quantitatively determine who is the best performer. Well-known metrics such as sensitivity, specificity, precision, or the area under the ROC curve, have been traditionally used. Tarca <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TarcaAComp13_22-3\" class=\"reference\"><a href=\"#cite_note-TarcaAComp13-22\">[22]<\/a><\/sup> introduced the use of sensitivity, specificity, and \u201cprioritization,\u201d together with False Positive Rate (FPR), where prioritization is a concept related to the rank of the target pathways for a given method. Zyla <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ZylaGene19_29-1\" class=\"reference\"><a href=\"#cite_note-ZylaGene19-29\">[29]<\/a><\/sup> have recently extended Tarca's work and recommended using a set of five different metrics: Sensitivity, False positive rate (FPR), Prioritization, Computational time, and Reproducibility. In their approach, reproducibility assigned a high score to methods that showed similar results for different datasets coming from the same condition\/disease and the same technical platform, but different authors\/labs.<sup id=\"rdp-ebb-cite_ref-ZylaGene19_29-2\" class=\"reference\"><a href=\"#cite_note-ZylaGene19-29\">[29]<\/a><\/sup> The chosen metrics are fundamental because no method is the best under all metrics, and every user should start by selecting a method according to the goals of their study, which may need a very sensitive method, a very specific method, or any other of the above-mentioned properties. As a consequence, benchmark studies must be clear regarding the metrics under which they are ranking the method performance<sup id=\"rdp-ebb-cite_ref-TarcaAComp13_22-4\" class=\"reference\"><a href=\"#cite_note-TarcaAComp13-22\">[22]<\/a><\/sup>, and their set of metrics should at least include the previously mentioned ones. Also, it makes no sense to declare success because a new method has better sensitivity than other methods that are known for a poor sensitivity; therefore, it is a logical consequence that any new methods must compare their sensitivity to sensitive methods, their specificity to specific methods, and so on.\n<\/p><p>A third issue is that validation procedures should not allow the authors to subjectively choose the methods that the authors want to compare to their new method, as it would be possible for the authors to choose those methods that they can outperform. One alternative to that is using well-established, independent, and comprehensive benchmark studies as a reference; then, when a new method appears, validation should be done by comparison to the top methods from such independent benchmarks. This practice is not common; as an exception, the authors of LEGO<sup id=\"rdp-ebb-cite_ref-DongLEGO16_32-0\" class=\"reference\"><a href=\"#cite_note-DongLEGO16-32\">[32]<\/a><\/sup> explicitly use the top five methods in Tarca <i>et al.'<\/i>s benchmark.<sup id=\"rdp-ebb-cite_ref-TarcaAComp13_22-5\" class=\"reference\"><a href=\"#cite_note-TarcaAComp13-22\">[22]<\/a><\/sup>\n<\/p><p>As a final thought, ensembles of methods have been suggested to give better results than any of the single methods.<sup id=\"rdp-ebb-cite_ref-AlhamdooshEasy17_33-0\" class=\"reference\"><a href=\"#cite_note-AlhamdooshEasy17-33\">[33]<\/a><\/sup> That implies that benchmark studies should not be limited to single tools but include comparisons to ensembles of tools as well. This approach has been followed by at least one benchmark study.<sup id=\"rdp-ebb-cite_ref-NaeemRigor12_21-2\" class=\"reference\"><a href=\"#cite_note-NaeemRigor12-21\">[21]<\/a><\/sup>\n<\/p><p>The previous analysis makes us think that future GSA benchmarks should include the most recent developments on benchmark theory, as well as be performed on more GSA methods, in order to extract more meaningful conclusions. In order to do this, we have also created a benchmarking platform, \u201c<a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/benchmarKING.html\" target=\"_blank\">GSA-BenchmarKING<\/a>,\u201d which is a repository of apps\/workflows\/pipelines that follow the above-mentioned good benchmarking practices and allow benchmarking of GSA software in an easy and automated way. Currently, GSA-BenchmarKING contains <a href=\"https:\/\/www.limswiki.org\/index.php\/Jupyter_Notebook\" title=\"Jupyter Notebook\" class=\"wiki-link\" data-key=\"26fd35430c10e009a142bbab5dbf617a\">Jupyter Notebooks<\/a> with full workflows for benchmarking GSA methods, and Shiny apps that allow benchmarking with the click of a few buttons. The initial benchmarking tools added to the repository were created by our students as an example, following the previous guidelines, but they are open software and can be permanently improved. The initial tools are focused on two types of methods: single-sample GSA (such as GSVA, Pathifier, SSGSEA, PLAGE, ZSCORE) and genomic-region GSA (such as GREAT, ChIP-Enrich, BroadEnrich, Enrichr, Seq2Pathways). All apps allow the user to define different gold standard datasets (or use ours), select the GSA methods to compare, and select the comparison metrics to plot. In addition, it is possible to keep adding more GSA methods to the app with the help of the community, as each app includes instructions for programmers interested in adding new methods. Given the large amount of GSA methods, more benchmarking tools are needed and welcomed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>The GSARefDB has been used to make some initial data exploration on the relationship between popularity and performance of GSA tools. Besides the observations highlighted in this article, such a database can also be the source for more in-depth research. For example, some limitations of our approach include (i) that some papers describe more than one method or platform, and (ii) that some methods are only cited for comparison purposes. We observe that such problems are the exception and not the rule, but future studies might want to take them into account and GSARefDB can still be used as a data source for it.\n<\/p><p>In general terms, it is evident that performance studies are still few, small, inconsistent, and dependent on the quality of the benchmarks; however, they tend to recommend tools different to the popular and friendly ones. Among the possible reasons for the discrepancies between popularity and performance, it has been suggested that software selection may not be entirely related to performance but to factors such as a preference for user-friendly platforms or user-friendly concepts or plots (for example, GSEA\u2019s \u201cenrichment plot\u201d). Also, due to the fact that the objective evaluation of the performance of the different GSA methods is a complicated and time-consuming issue, or to the fact that new methods need more time to be accepted. In network analysis, popularity can be explained by the \u201crich-gets-richer\u201d (popular-gets-more-popular) effect. Using concepts of the \u201cconsumer behavior\u201d field, software selection can be studied as the choice of a popular brand, that is, variables such as confidence in experience (the respect to the researcher\/institution associated to the software), social acceptance and personal image (following the software that everyone else is using), or consumer loyalty (after some time, we are attached to our software and not interested to change).<sup id=\"rdp-ebb-cite_ref-34\" class=\"reference\"><a href=\"#cite_note-34\">[34]<\/a><\/sup> In a recent book, Barabasi<sup id=\"rdp-ebb-cite_ref-Barab.C3.A1siTheForm18_35-0\" class=\"reference\"><a href=\"#cite_note-Barab.C3.A1siTheForm18-35\">[35]<\/a><\/sup> suggested that popularity and quality usually go together in situations where performance is clearly measurable. Otherwise, popularity can't be equated to quality. Our study agrees with such an idea. We have found that the most popular GSA software is different from the best performing GSA software, and that thorough performance evaluation is still a pending assignment for bioinformaticians.\n<\/p><p>However, in the specific case of GSA, there is a complementary explanatory hypothesis: Besides all the progress in GSA theory leading to better-performing methods, at the end of the pipeline, the users usually extract the evidence that they consider relevant from the resulting rank of gene sets, choosing the pathways or GO terms of their interest and ignoring the rest of the ranking, and, therefore, any differences between methods regarding lower <i>p<\/i>-values, prioritization order, and so on, become less important. It is said that in the step of gene set selection from the final gene set ranking, researchers bring \"context\" to the results but, this way, their subjectivity may be projected to the study. One way out of this is to stimulate the research on \u201ccontext-based GSA.\u201d For example, a recent effort called contextTRAP<sup id=\"rdp-ebb-cite_ref-LeePrior16_36-0\" class=\"reference\"><a href=\"#cite_note-LeePrior16-36\">[36]<\/a><\/sup> combines an impact score (from pathway analysis) with a context score (from text mining information that supports that the pathway is relevant to the context of the experiment). Bayesian approximations to GSA using data as context need to be more developed, as any other method that studies the final gene set ranking as a whole.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Given that popular methods are not necessarily the best, bioinformatics software users should not only be guided by popularity but mainly by performance studies. However, performance studies, in time, must be guided by the general guidelines that we have discussed, that is, researchers should only follow the most convincing benchmark procedures. Such strict recommendations are problematic because performance studies are few, low-coverage, and have a variable quality; therefore, we need more open tools to dynamically review popularity and performance of bioinformatics software, such as those introduced here.\n<\/p><p>Based on the previous results and discussion, we argue that the functional interpretation field would benefit from:\n<\/p><p>1. having more information and discussion regarding the nature and scope of the existing functional interpretation methods of omics data, as well as more teaching of new and sophisticated methods in bioinformatics courses, more guidelines for selection of a tool, and more popularization of the functional interpretation shortcomings. Also, deeper discussions of the selection of GSA methods for all biomedical papers should be requested.\n<\/p><p>2. the permanent evaluation of GSA methods, including better gold standards, an increasing number of comprehensive comparison studies, and better benchmarking practices. The fact that two prestigious computational biology journals have recently created a special edition and a collection specialized on benchmarks are welcome steps in that direction.<sup id=\"rdp-ebb-cite_ref-VitekBench20_37-0\" class=\"reference\"><a href=\"#cite_note-VitekBench20-37\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PLoSComp18_38-0\" class=\"reference\"><a href=\"#cite_note-PLoSComp18-38\">[38]<\/a><\/sup>\n<\/p><p>3. paying attention to reproducibility and offering open code in code-sharing platforms (such as GitHub), containers with the specific software and library versions used in their work (such as Docker), notebooks (such as RStudio and Jupyter) including scripts with a detailed explanation of their methods, and other strategies to allow reproducibility.\n<\/p><p>4. the creation of a culture of functional re-analysis of existing data using new GSA methods, as well as the computational tools to functionally re-analyze existing omics datasets in a streamlined manner.\n<\/p><p>5. more rigorous validation procedures for GSA tools. Also, bioinformaticians should get as much training on scientific validation methods and tools as they get on using and building bioinformatics tools.\n<\/p><p>Proper tool selection is fundamental for generating high-quality results in all scientific fields. This paper suggests that tool performance and tool selection studies, via the popularity-performance evaluation based on an exhaustive reference database, is a methodology that should be followed up, to keep track of the evolution of the tool selection issues in a scientific field. We have also introduced examples of popularity- and performance-measuring software that could help make such studies easier. The reader is here invited to keep following our work for the GSA field, including <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/gsarefdb.html\" target=\"_blank\">GSARefDB<\/a> and <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/benchmarKING.html\" target=\"_blank\">GSA-BenchmarKING<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Definitions\">Definitions<\/span><\/h3>\n<p>The concepts involved in this study have been defined in several different ways in the literature. For example, the field under study has been called \"pathway analysis,\"<sup id=\"rdp-ebb-cite_ref-KhatriTenYears12_5-1\" class=\"reference\"><a href=\"#cite_note-KhatriTenYears12-5\">[5]<\/a><\/sup> \"enrichment analysis,\"<sup id=\"rdp-ebb-cite_ref-Bayerlov.C3.A1Comp15_23-1\" class=\"reference\"><a href=\"#cite_note-Bayerlov.C3.A1Comp15-23\">[23]<\/a><\/sup> \"gene set analysis,\"<sup id=\"rdp-ebb-cite_ref-MoraGene20_2-2\" class=\"reference\"><a href=\"#cite_note-MoraGene20-2\">[2]<\/a><\/sup> \"functional enrichment analysis,\"<sup id=\"rdp-ebb-cite_ref-YuCluster12_12-1\" class=\"reference\"><a href=\"#cite_note-YuCluster12-12\">[12]<\/a><\/sup> \u201cgene-annotation enrichment analysis,\u201d<sup id=\"rdp-ebb-cite_ref-HuangBioinfo09_4-2\" class=\"reference\"><a href=\"#cite_note-HuangBioinfo09-4\">[4]<\/a><\/sup> and other terms, by different authors. At the same time, the term \"gene set analysis\" has been used to describe the entire field<sup id=\"rdp-ebb-cite_ref-MoraGene20_2-3\" class=\"reference\"><a href=\"#cite_note-MoraGene20-2\">[2]<\/a><\/sup>, or just the group of ORA and FCS methods (in opposition to methods including pathway or network topology)<sup id=\"rdp-ebb-cite_ref-Bayerlov.C3.A1Comp15_23-2\" class=\"reference\"><a href=\"#cite_note-Bayerlov.C3.A1Comp15-23\">[23]<\/a><\/sup>, or even one specific tool.<sup id=\"rdp-ebb-cite_ref-EfroniIdent07_39-0\" class=\"reference\"><a href=\"#cite_note-EfroniIdent07-39\">[39]<\/a><\/sup> Finally, the term \u201cpathway analysis\u201d has also been used to both describe the entire field<sup id=\"rdp-ebb-cite_ref-KhatriTenYears12_5-2\" class=\"reference\"><a href=\"#cite_note-KhatriTenYears12-5\">[5]<\/a><\/sup>, or just the group of methods that include pathway topology.<sup id=\"rdp-ebb-cite_ref-Bayerlov.C3.A1Comp15_23-3\" class=\"reference\"><a href=\"#cite_note-Bayerlov.C3.A1Comp15-23\">[23]<\/a><\/sup> For this reason, we have added the following summary of the definitions used in this study for the above-mentioned and other relevant terms.\n<\/p><p><b>benchmark study<\/b>: A systematic comparison between computational methods, in which all of them are applied to a gold standard dataset and the success of their gene set predictions are summarized in terms of quantitative metrics (such as sensitivity, specificity, and others).\n<\/p><p><b>functional class scoring (FCS)<\/b>: A subset of GSA methods in which the values of a gene-level statistic for all genes in the experiment are aggregated into a gene-set-level statistics<sup id=\"rdp-ebb-cite_ref-KhatriTenYears12_5-3\" class=\"reference\"><a href=\"#cite_note-KhatriTenYears12-5\">[5]<\/a><\/sup>, and gene set enrichment is computed in terms of the significance of such gene-set-level statistic. FCS methods start from a quantitative ranking of the gene-level statistic for all genes under analysis (in contrast to ORA, which only uses a list of differentially expressed genes). Some popular FCS methods find if the relative position of a gene set in the ranking of all genes is shifted to the top or the bottom of the ranking. For example, the WRS test compares the distribution of ranks of the genes in a gene set to the distribution of ranks of the genes in the complement to the gene set, while the KS test compares the ranks of genes in a gene set to a uniform distribution.<sup id=\"rdp-ebb-cite_ref-Bayerlov.C3.A1Comp15_23-4\" class=\"reference\"><a href=\"#cite_note-Bayerlov.C3.A1Comp15-23\">[23]<\/a><\/sup>\n<\/p><p><b>gene set analysis (GSA)<\/b>: GSA methods have been defined as a group of \"methods that aim to identify the pathways that are significantly impacted in a condition under study\"<sup id=\"rdp-ebb-cite_ref-MitreaMeth13_6-1\" class=\"reference\"><a href=\"#cite_note-MitreaMeth13-6\">[6]<\/a><\/sup> or as \"tests which aim to detect pathways significantly enriched between two experimental conditions.\"<sup id=\"rdp-ebb-cite_ref-Bayerlov.C3.A1Comp15_23-5\" class=\"reference\"><a href=\"#cite_note-Bayerlov.C3.A1Comp15-23\">[23]<\/a><\/sup> More specifically, GSA is an annotation-based approach that statistically compares experimental results to an annotated database in order to transform gene-level results into gene-set-level results. For example, a query gene set (a list of differentially expressed genes, or a rank of all gene\u2019s fold change) is mapped to a gene set reference database, using a particular statistical method, in order to explain the experimental results as a rank of significantly impacted pathways, functionally related gene sets, or ontology terms.\n<\/p><p><b>gold standard<\/b>: A \"perfect gold standard\" would be an error-free dataset that can be used as a synonym for truth (in our case, an omics dataset associated to a true ranking of pathways); however, in practice, we are limited to use \"imperfect\" or \"alloyed gold standards,\" which are datasets confidently linked to the truth, but not necessarily datasets lacking error.<sup id=\"rdp-ebb-cite_ref-WacholderValid93_40-0\" class=\"reference\"><a href=\"#cite_note-WacholderValid93-40\">[40]<\/a><\/sup>\n<\/p><p><b>network interaction (NI)<\/b>: A subset of GSA methods that not only includes the given gene sets but also the gene products that interact with the members of such gene sets when located on top of an interaction or functional annotation network.\n<\/p><p><b>over-representation analysis (ORA)<\/b>: A subset of GSA methods based on comparing a list of query genes (for example, up- or down-regulated genes) to a list of genes in a class or gene set, using a statistical test that detects over-representation. ORA \"statistically evaluates the fraction of genes in a particular pathway found among the set of genes showing changes in expression.\"<sup id=\"rdp-ebb-cite_ref-KhatriTenYears12_5-4\" class=\"reference\"><a href=\"#cite_note-KhatriTenYears12-5\">[5]<\/a><\/sup>\n<\/p><p><b>pathway-topology-based (PT)<\/b>: A subset of GSA methods that weighs enrichment scores according to the position of a gene in a pathway. Only applies to pathway data and not to other types of gene sets.\n<\/p><p><b>performance<\/b>: The value of a quantitative property (when compared to alternative methods or tools) that measures the agreement between the method's output and either empirical data, simulated data, or the output of another method.\n<\/p><p><b>popularity<\/b>: The frequency of use of a method or tool among members of a community.\n<\/p><p><b>simulation study<\/b>: A systematic comparison of computational methods based on building artificial datasets that possess the properties that we specify for them.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Construction_of_the_database\">Construction of the database<\/span><\/h3>\n<p>The Gene Set Analysis Reference Database (GSARefDB) was built from the following sources:\n<\/p><p>1. Google and PubMed searches of keywords such as: \"pathway analysis,\" \"gene set analysis,\" and \"functional enrichment\" (approx. 10% of the records)\n<\/p><p>2. Cross-references from all the collected papers and reviews (approx. 50% of the records)\n<\/p><p>3. Email alerts received from NCBI and selected journals (approx. 40% of the records)\n<\/p><p>The information was classified into: (i) generic methods\/software\/platforms, usually dealing with mRNA datasets; (ii) reviews\/benchmark studies; (iii) genomic GSA, which includes GSA applied to enrichment of genomic regions (such as those coming from ChIP-seq, SNP, and methylation experiments); and (iv) ncRNA GSA, which includes methods dealing with miRNA and lncRNA datasets. All information in the database was manually extracted from the papers. Numbers of citations were extracted from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/scholar.google.com)\" target=\"_blank\">Google Scholar<\/a>. Only methods that associate omics data to annotated gene sets were included (see all types of included methods in the prior Fig. 1b). Bioinformatics methods that associate omics data to newly generated modules on a biological network were not included. GSARefDB was built both as an excel sheet and as a Shiny application (Figure 2a).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Xie_BMCBioinfo21_22.png\" class=\"image wiki-link\" data-key=\"1879bfb7353fdd018a8d392495258b64\"><img alt=\"Fig2 Xie BMCBioinfo21 22.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/39\/Fig2_Xie_BMCBioinfo21_22.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Screenshots of our tools for popularity and performance analysis of the GSA field. <b>a<\/b> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/gsarefdb.html\" target=\"_blank\">GSARefDB<\/a>: A screenshot of the R\/Shiny interface to GSARefDB, showing the options of searching by year, tool name, paper\u2019s first author, title, type of GSA, and programming language. <b>b<\/b> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/benchmarKING.html\" target=\"_blank\">GSA BenchmarKING<\/a>: One jupyter notebook containing an R workflow for benchmarking single-sample GSA methods, and one shiny app with the same purpose. Both tools display sensitivity, specificity, and precision plots for all the methods under study.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Descriptive_statistics\">Descriptive statistics<\/span><\/h3>\n<p>Plots of summary statistics were generated using the \u201cggplot2\u201d R package. The R code is open and can be <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/antonio-mora\/paperCode\/blob\/master\/2019_Mora_Popularity_versus_Performance.R\" target=\"_blank\">found here<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Popularity_rankings\">Popularity rankings<\/span><\/h3>\n<p>Popularity rankings in GSARefDB are constructed on a per paper basis, not per-method or per-tool. In order to build popularity rankings of methods or tools that are presented in multiple papers, we use the citation count of the most cited paper for that tool.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Performance_study\">Performance study<\/span><\/h3>\n<p>The scientific validation approaches followed by the top 153 GSA tool papers in our database were manually reviewed (see Additional file 1: Tab 6). Validation, as performance, was defined as the success of a method on getting better scores than rival methods for a specific quantitative property that measures the agreement between the method's output and either empirical data, simulated data, or the output of another method. Our definition of validation does not include:\n<\/p><p>(i) examples of application of the method followed by highlighting the reasonableness of the results (without comparing to other methods);\n<\/p><p>(ii) arguments (usually statistical) stating that new assumptions are better than old ones, without any comparison to empirical or properly simulated data; or\n<\/p><p>(iii) comparisons of capabilities between old and new software (such as implementing other algorithms or databases).\n<\/p><p>Detailed procedures followed during the benchmark and simulation studies are explained in Additional file 2.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Construction_of_performance-measuring_software\">Construction of performance-measuring software<\/span><\/h3>\n<p>The \u201cGSA-BenchmarKING\u201d repository (see Figure 2b and the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/benchmarKING.html\" target=\"_blank\">GSA BenchmarKING page<\/a>) was created to store and share different tools to measure GSA method performance. For a benchmarking software to be accepted into the repository, it should: (i) be open software (ideally, a Jupyter Notebook, RStudio R Notebook, or Shiny app); (ii) have a clear reason for selecting the GSA methods included; for example, because all of them belong to the same type of methods; (iii) include both a gold standard dataset and options to upload user-selected gold standard datasets; (iv) include either a list of target pathways linked to the gold standard dataset or disease relevance scores per pathway for the diseases related to the gold standard; (v) give the user the option of selecting different benchmarking metrics (as a minimum, precision\/sensitivity, prioritization, and specificity\/FPR); (vi) give the possibility of selecting ensemble results; and (vii) give the possibility of adding new GSA methods to the code in the future.\n<\/p><p>The \u201cGSARefDB,\u201d \u201css-shiny,\u201d and \u201cgr-shiny\u201d apps were built using the \u201cShiny\u201d package. \u201cGSARefDB\u201d and \u201css-shiny\u201d were built using R 3.6.2, while \u201cgr-shiny\u201d was built using R 4.0.0. Open code can be accessed at GitHub <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/gsa-central\/gsarefdb\" target=\"_blank\">here<\/a>, <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/mora-lab\/ss-shiny\" target=\"_blank\">here<\/a>, and <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/mora-lab\/gr-shiny\" target=\"_blank\">here<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_information\">Supplementary information<\/span><\/h2>\n<ul><li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1186%2Fs12859-021-04124-5\/MediaObjects\/12859_2021_4124_MOESM1_ESM.xlsx\" target=\"_blank\">Additional file 1<\/a>: GSA Reference DB<\/li>\n<li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1186%2Fs12859-021-04124-5\/MediaObjects\/12859_2021_4124_MOESM2_ESM.docx\" target=\"_blank\">Additional file 2<\/a>: Detailed Performance Study<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors thank the support of both colleagues and students at Guangzhou Medical University and Guangzhou Institutes of Biomedicine and Health (Chinese Academy of Sciences). We also thank our two anonymous reviewers for their contributions.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Contributions\">Contributions<\/span><\/h3>\n<p>CSX wrote the Shiny apps for GSARefDB and ss-shiny, and generated the figures. SJ programmed the Shiny app for gr-shiny. AM designed the project, built the reference database, and wrote the paper. All authors read and approved the final version of the paper.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Availability_of_data_and_materials\">Availability of data and materials<\/span><\/h3>\n<p>All datasets generated for this study are included in the supplementary material. Updated versions of the datasets will be available at the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/gsarefdb.html\" target=\"_blank\">GSARefDB<\/a> and <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/benchmarKING.html\" target=\"_blank\">GSA BenchmarKING<\/a> pages.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work was funded by the Joint School of Life Sciences, Guangzhou Medical University and Guangzhou Institutes of Biomedicine and Health (Chinese Academy of Sciences). SJ is also funded by the China Postdoctoral Science Foundation Grant No. 2019M652847. The funders had no role in the design of the study, data collection, data analysis, interpretation of results, or writing of the paper.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-DixsonRetract14-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DixsonRetract14_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dixson, L.; Walter, H.; Schneider, M. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4169929\" target=\"_blank\">\"Retraction for Dixson et al., Identification of gene ontologies linked to prefrontal-hippocampal functional coupling in the human brain\"<\/a>. <i>Proceedings of the National Academy of Sciences of the United States of America<\/i> <b>111<\/b> (37): 13582. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1073%2Fpnas.1414905111\" target=\"_blank\">10.1073\/pnas.1414905111<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4169929\/\" target=\"_blank\">PMC4169929<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25197092\" target=\"_blank\">25197092<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4169929\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4169929<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Retraction+for+Dixson+et+al.%2C+Identification+of+gene+ontologies+linked+to+prefrontal-hippocampal+functional+coupling+in+the+human+brain&rft.jtitle=Proceedings+of+the+National+Academy+of+Sciences+of+the+United+States+of+America&rft.aulast=Dixson%2C+L.%3B+Walter%2C+H.%3B+Schneider%2C+M.+et+al.&rft.au=Dixson%2C+L.%3B+Walter%2C+H.%3B+Schneider%2C+M.+et+al.&rft.date=2014&rft.volume=111&rft.issue=37&rft.pages=13582&rft_id=info:doi\/10.1073%2Fpnas.1414905111&rft_id=info:pmc\/PMC4169929&rft_id=info:pmid\/25197092&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4169929&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoraGene20-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MoraGene20_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-MoraGene20_2-1\">2.1<\/a><\/sup> <sup><a href=\"#cite_ref-MoraGene20_2-2\">2.2<\/a><\/sup> <sup><a href=\"#cite_ref-MoraGene20_2-3\">2.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mora, A. (2020). \"Gene set analysis methods for the functional interpretation of non-mRNA data-Genomic range and ncRNA data\". <i>Briefings in Bioinformatics<\/i> <b>21<\/b> (5): 1495-1508. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbz090\" target=\"_blank\">10.1093\/bib\/bbz090<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31612220\" target=\"_blank\">31612220<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+set+analysis+methods+for+the+functional+interpretation+of+non-mRNA+data-Genomic+range+and+ncRNA+data&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Mora%2C+A.&rft.au=Mora%2C+A.&rft.date=2020&rft.volume=21&rft.issue=5&rft.pages=1495-1508&rft_id=info:doi\/10.1093%2Fbib%2Fbbz090&rft_id=info:pmid\/31612220&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TavazoieSystem99-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TavazoieSystem99_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-TavazoieSystem99_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tavazoie, S.; Hughes, J.D.; Campbell, M.J. et al. (1999). \"Systematic determination of genetic network architecture\". <i>Nature Genetics<\/i> <b>22<\/b> (3): 281\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2F10343\" target=\"_blank\">10.1038\/10343<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10391217\" target=\"_blank\">10391217<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Systematic+determination+of+genetic+network+architecture&rft.jtitle=Nature+Genetics&rft.aulast=Tavazoie%2C+S.%3B+Hughes%2C+J.D.%3B+Campbell%2C+M.J.+et+al.&rft.au=Tavazoie%2C+S.%3B+Hughes%2C+J.D.%3B+Campbell%2C+M.J.+et+al.&rft.date=1999&rft.volume=22&rft.issue=3&rft.pages=281%E2%80%935&rft_id=info:doi\/10.1038%2F10343&rft_id=info:pmid\/10391217&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuangBioinfo09-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HuangBioinfo09_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-HuangBioinfo09_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-HuangBioinfo09_4-2\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Huang, D.W.; Sherman, B.T.; Lempicki, R.A. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2615629\" target=\"_blank\">\"Bioinformatics enrichment tools: Paths toward the comprehensive functional analysis of large gene lists\"<\/a>. <i>Nucleic Acids Research<\/i> <b>37<\/b> (1): 1-13. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkn923\" target=\"_blank\">10.1093\/nar\/gkn923<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2615629\/\" target=\"_blank\">PMC2615629<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19033363\" target=\"_blank\">19033363<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2615629\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2615629<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+enrichment+tools%3A+Paths+toward+the+comprehensive+functional+analysis+of+large+gene+lists&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Huang%2C+D.W.%3B+Sherman%2C+B.T.%3B+Lempicki%2C+R.A.&rft.au=Huang%2C+D.W.%3B+Sherman%2C+B.T.%3B+Lempicki%2C+R.A.&rft.date=2009&rft.volume=37&rft.issue=1&rft.pages=1-13&rft_id=info:doi\/10.1093%2Fnar%2Fgkn923&rft_id=info:pmc\/PMC2615629&rft_id=info:pmid\/19033363&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2615629&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhatriTenYears12-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KhatriTenYears12_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-KhatriTenYears12_5-1\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-KhatriTenYears12_5-2\">5.2<\/a><\/sup> <sup><a href=\"#cite_ref-KhatriTenYears12_5-3\">5.3<\/a><\/sup> <sup><a href=\"#cite_ref-KhatriTenYears12_5-4\">5.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khatri, P.; Sirota, M.; Butte, A.J. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3285573\" target=\"_blank\">\"Ten years of pathway analysis: Current approaches and outstanding challenges\"<\/a>. <i>PLoS Computational Biology<\/i> <b>8<\/b> (2): e1002375. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1002375\" target=\"_blank\">10.1371\/journal.pcbi.1002375<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3285573\/\" target=\"_blank\">PMC3285573<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22383865\" target=\"_blank\">22383865<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3285573\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3285573<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ten+years+of+pathway+analysis%3A+Current+approaches+and+outstanding+challenges&rft.jtitle=PLoS+Computational+Biology&rft.aulast=Khatri%2C+P.%3B+Sirota%2C+M.%3B+Butte%2C+A.J.&rft.au=Khatri%2C+P.%3B+Sirota%2C+M.%3B+Butte%2C+A.J.&rft.date=2012&rft.volume=8&rft.issue=2&rft.pages=e1002375&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1002375&rft_id=info:pmc\/PMC3285573&rft_id=info:pmid\/22383865&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3285573&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MitreaMeth13-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MitreaMeth13_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-MitreaMeth13_6-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mitrea, C.; Taghavi, Z.; Bokanizad, B. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3794382\" target=\"_blank\">\"Methods and approaches in the topology-based analysis of biological pathways\"<\/a>. <i>Frontiers in Physiology<\/i> <b>4<\/b>: 278. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffphys.2013.00278\" target=\"_blank\">10.3389\/fphys.2013.00278<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3794382\/\" target=\"_blank\">PMC3794382<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24133454\" target=\"_blank\">24133454<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3794382\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3794382<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Methods+and+approaches+in+the+topology-based+analysis+of+biological+pathways&rft.jtitle=Frontiers+in+Physiology&rft.aulast=Mitrea%2C+C.%3B+Taghavi%2C+Z.%3B+Bokanizad%2C+B.+et+al.&rft.au=Mitrea%2C+C.%3B+Taghavi%2C+Z.%3B+Bokanizad%2C+B.+et+al.&rft.date=2013&rft.volume=4&rft.pages=278&rft_id=info:doi\/10.3389%2Ffphys.2013.00278&rft_id=info:pmc\/PMC3794382&rft_id=info:pmid\/24133454&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3794382&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GSARedDB-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GSARedDB_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Mora, A. (26 March 2021). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gsa-central.github.io\/gsarefdb.html\" target=\"_blank\">\"GSARefDB\"<\/a>. <i>GSA Central<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/gsa-central.github.io\/gsarefdb.html\" target=\"_blank\">https:\/\/gsa-central.github.io\/gsarefdb.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GSARefDB&rft.atitle=GSA+Central&rft.aulast=Mora%2C+A.&rft.au=Mora%2C+A.&rft.date=26+March+2021&rft_id=https%3A%2F%2Fgsa-central.github.io%2Fgsarefdb.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuangDAVID07-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HuangDAVID07_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-HuangDAVID07_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Huang, D.W.; Sherman, B.T.; Tan, Q. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1933169\" target=\"_blank\">\"DAVID Bioinformatics Resources: Expanded annotation database and novel algorithms to better extract biology from large gene lists\"<\/a>. <i>Nucleic Acids Research<\/i> <b>35<\/b> (Web Server Issue): W169-75. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkm415\" target=\"_blank\">10.1093\/nar\/gkm415<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1933169\/\" target=\"_blank\">PMC1933169<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17576678\" target=\"_blank\">17576678<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1933169\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1933169<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DAVID+Bioinformatics+Resources%3A+Expanded+annotation+database+and+novel+algorithms+to+better+extract+biology+from+large+gene+lists&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Huang%2C+D.W.%3B+Sherman%2C+B.T.%3B+Tan%2C+Q.+et+al.&rft.au=Huang%2C+D.W.%3B+Sherman%2C+B.T.%3B+Tan%2C+Q.+et+al.&rft.date=2007&rft.volume=35&rft.issue=Web+Server+Issue&rft.pages=W169-75&rft_id=info:doi\/10.1093%2Fnar%2Fgkm415&rft_id=info:pmc\/PMC1933169&rft_id=info:pmid\/17576678&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1933169&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SubramanianGene05-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SubramanianGene05_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Subramanian, A.; Tamayo, P.; Mootha, V.K. et al. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1239896\" target=\"_blank\">\"Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles\"<\/a>. <i>Proceedings of the National Academy of Sciences of the United States of America<\/i> <b>102<\/b> (43): 15545-50. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1073%2Fpnas.0506580102\" target=\"_blank\">10.1073\/pnas.0506580102<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1239896\/\" target=\"_blank\">PMC1239896<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16199517\" target=\"_blank\">16199517<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1239896\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1239896<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+set+enrichment+analysis%3A+a+knowledge-based+approach+for+interpreting+genome-wide+expression+profiles&rft.jtitle=Proceedings+of+the+National+Academy+of+Sciences+of+the+United+States+of+America&rft.aulast=Subramanian%2C+A.%3B+Tamayo%2C+P.%3B+Mootha%2C+V.K.+et+al.&rft.au=Subramanian%2C+A.%3B+Tamayo%2C+P.%3B+Mootha%2C+V.K.+et+al.&rft.date=2005&rft.volume=102&rft.issue=43&rft.pages=15545-50&rft_id=info:doi\/10.1073%2Fpnas.0506580102&rft_id=info:pmc\/PMC1239896&rft_id=info:pmid\/16199517&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1239896&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DraghiciASys07-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DraghiciASys07_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Draghici, S.; Khatri, P.; Laurentiu, A. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1987343\" target=\"_blank\">\"A systems biology approach for pathway level analysis\"<\/a>. <i>Genome Research<\/i> <b>17<\/b> (10): 1537\u201345. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.6202607\" target=\"_blank\">10.1101\/gr.6202607<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1987343\/\" target=\"_blank\">PMC1987343<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17785539\" target=\"_blank\">17785539<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1987343\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1987343<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+systems+biology+approach+for+pathway+level+analysis&rft.jtitle=Genome+Research&rft.aulast=Draghici%2C+S.%3B+Khatri%2C+P.%3B+Laurentiu%2C+A.+et+al.&rft.au=Draghici%2C+S.%3B+Khatri%2C+P.%3B+Laurentiu%2C+A.+et+al.&rft.date=2007&rft.volume=17&rft.issue=10&rft.pages=1537%E2%80%9345&rft_id=info:doi\/10.1101%2Fgr.6202607&rft_id=info:pmc\/PMC1987343&rft_id=info:pmid\/17785539&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1987343&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-.C5.90szASnap19-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-.C5.90szASnap19_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">\u0150sz, \u00c1.; Pongor, L.S.; Szirmai, D. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6585384\" target=\"_blank\">\"A snapshot of 3649 Web-based services published between 1994 and 2017 shows a decrease in availability after two years\"<\/a>. <i>Briefings in Bioinformatics<\/i> <b>20<\/b> (3): 1004-1010. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbx159\" target=\"_blank\">10.1093\/bib\/bbx159<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6585384\/\" target=\"_blank\">PMC6585384<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29228189\" target=\"_blank\">29228189<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6585384\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6585384<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+snapshot+of+3649+Web-based+services+published+between+1994+and+2017+shows+a+decrease+in+availability+after+two+years&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=%C5%90sz%2C+%C3%81.%3B+Pongor%2C+L.S.%3B+Szirmai%2C+D.+et+al.&rft.au=%C5%90sz%2C+%C3%81.%3B+Pongor%2C+L.S.%3B+Szirmai%2C+D.+et+al.&rft.date=2019&rft.volume=20&rft.issue=3&rft.pages=1004-1010&rft_id=info:doi\/10.1093%2Fbib%2Fbbx159&rft_id=info:pmc\/PMC6585384&rft_id=info:pmid\/29228189&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6585384&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YuCluster12-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-YuCluster12_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-YuCluster12_12-1\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yu, G.; Wang,L.-G.; Hanf, Y. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3339379\" target=\"_blank\">\"clusterProfiler: an R package for comparing biological themes among gene clusters\"<\/a>. <i>OMICS<\/i> <b>16<\/b> (5): 284\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fomi.2011.0118\" target=\"_blank\">10.1089\/omi.2011.0118<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3339379\/\" target=\"_blank\">PMC3339379<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22455463\" target=\"_blank\">22455463<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3339379\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3339379<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=clusterProfiler%3A+an+R+package+for+comparing+biological+themes+among+gene+clusters&rft.jtitle=OMICS&rft.aulast=Yu%2C+G.%3B+Wang%2CL.-G.%3B+Hanf%2C+Y.+et+al.&rft.au=Yu%2C+G.%3B+Wang%2CL.-G.%3B+Hanf%2C+Y.+et+al.&rft.date=2012&rft.volume=16&rft.issue=5&rft.pages=284%E2%80%937&rft_id=info:doi\/10.1089%2Fomi.2011.0118&rft_id=info:pmc\/PMC3339379&rft_id=info:pmid\/22455463&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3339379&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KuleshovEnrichr16-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KuleshovEnrichr16_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kuleshov, M.V.; Jones, M.R.; Rouillard, A.D. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987924\" target=\"_blank\">\"Enrichr: A comprehensive gene set enrichment analysis web server 2016 update\"<\/a>. <i>Nucleic Acids Research<\/i> <b>44<\/b> (W1): W90\u20147. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkw377\" target=\"_blank\">10.1093\/nar\/gkw377<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4987924\/\" target=\"_blank\">PMC4987924<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27141961\" target=\"_blank\">27141961<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987924\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987924<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enrichr%3A+A+comprehensive+gene+set+enrichment+analysis+web+server+2016+update&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Kuleshov%2C+M.V.%3B+Jones%2C+M.R.%3B+Rouillard%2C+A.D.+et+al.&rft.au=Kuleshov%2C+M.V.%3B+Jones%2C+M.R.%3B+Rouillard%2C+A.D.+et+al.&rft.date=2016&rft.volume=44&rft.issue=W1&rft.pages=W90%E2%80%947&rft_id=info:doi\/10.1093%2Fnar%2Fgkw377&rft_id=info:pmc\/PMC4987924&rft_id=info:pmid\/27141961&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4987924&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YoungGene10-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YoungGene10_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Young, M.D.; Wakefield, M.J.; Smyth, G.K. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2872874\" target=\"_blank\">\"Gene ontology analysis for RNA-seq: accounting for selection bias\"<\/a>. <i>Genome Biology<\/i> <b>11<\/b> (2): R14. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2010-11-2-r14\" target=\"_blank\">10.1186\/gb-2010-11-2-r14<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2872874\/\" target=\"_blank\">PMC2872874<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20132535\" target=\"_blank\">20132535<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2872874\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2872874<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+ontology+analysis+for+RNA-seq%3A+accounting+for+selection+bias&rft.jtitle=Genome+Biology&rft.aulast=Young%2C+M.D.%3B+Wakefield%2C+M.J.%3B+Smyth%2C+G.K.+et+al.&rft.au=Young%2C+M.D.%3B+Wakefield%2C+M.J.%3B+Smyth%2C+G.K.+et+al.&rft.date=2010&rft.volume=11&rft.issue=2&rft.pages=R14&rft_id=info:doi\/10.1186%2Fgb-2010-11-2-r14&rft_id=info:pmc\/PMC2872874&rft_id=info:pmid\/20132535&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2872874&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BindeaClue09-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BindeaClue09_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bindea, G.; Mlecnik, B.; Hackl, H. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2666812\" target=\"_blank\">\"ClueGO: A Cytoscape plug-in to decipher functionally grouped gene ontology and pathway annotation networks\"<\/a>. <i>Bioinformatics<\/i> <b>25<\/b> (8): 1091\u20133. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp101\" target=\"_blank\">10.1093\/bioinformatics\/btp101<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2666812\/\" target=\"_blank\">PMC2666812<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19237447\" target=\"_blank\">19237447<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2666812\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2666812<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ClueGO%3A+A+Cytoscape+plug-in+to+decipher+functionally+grouped+gene+ontology+and+pathway+annotation+networks&rft.jtitle=Bioinformatics&rft.aulast=Bindea%2C+G.%3B+Mlecnik%2C+B.%3B+Hackl%2C+H.+et+al.&rft.au=Bindea%2C+G.%3B+Mlecnik%2C+B.%3B+Hackl%2C+H.+et+al.&rft.date=2009&rft.volume=25&rft.issue=8&rft.pages=1091%E2%80%933&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp101&rft_id=info:pmc\/PMC2666812&rft_id=info:pmid\/19237447&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2666812&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GiannoulatouVerif14-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GiannoulatouVerif14_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Giannoulatouet, E.; Park, S.-H.; Humphreys, D.T. al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4290646\" target=\"_blank\">\"Verification and validation of bioinformatics software without a gold standard: A case study of BWA and Bowtie\"<\/a>. <i>BMC Bioinformatics<\/i> <b>15<\/b> (Suppl. 16): S15. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-15-S16-S15\" target=\"_blank\">10.1186\/1471-2105-15-S16-S15<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4290646\/\" target=\"_blank\">PMC4290646<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25521810\" target=\"_blank\">25521810<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4290646\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4290646<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Verification+and+validation+of+bioinformatics+software+without+a+gold+standard%3A+A+case+study+of+BWA+and+Bowtie&rft.jtitle=BMC+Bioinformatics&rft.aulast=Giannoulatouet%2C+E.%3B+Park%2C+S.-H.%3B+Humphreys%2C+D.T.+al.&rft.au=Giannoulatouet%2C+E.%3B+Park%2C+S.-H.%3B+Humphreys%2C+D.T.+al.&rft.date=2014&rft.volume=15&rft.issue=Suppl.+16&rft.pages=S15&rft_id=info:doi\/10.1186%2F1471-2105-15-S16-S15&rft_id=info:pmc\/PMC4290646&rft_id=info:pmid\/25521810&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4290646&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CurtisPath05-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CurtisPath05_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Curtis, R.K.; Oresic, M.; Vidal-Puig, A. (2005). \"Pathways to the analysis of microarray data\". <i>Trends in Biotechnology<\/i> <b>23<\/b> (8): 429-35. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.tibtech.2005.05.011\" target=\"_blank\">10.1016\/j.tibtech.2005.05.011<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15950303\" target=\"_blank\">15950303<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pathways+to+the+analysis+of+microarray+data&rft.jtitle=Trends+in+Biotechnology&rft.aulast=Curtis%2C+R.K.%3B+Oresic%2C+M.%3B+Vidal-Puig%2C+A.&rft.au=Curtis%2C+R.K.%3B+Oresic%2C+M.%3B+Vidal-Puig%2C+A.&rft.date=2005&rft.volume=23&rft.issue=8&rft.pages=429-35&rft_id=info:doi\/10.1016%2Fj.tibtech.2005.05.011&rft_id=info:pmid\/15950303&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TarcaDown12-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TarcaDown12_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tarca, A.L.; Draghici, S.; Bhatti, G. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3443069\" target=\"_blank\">\"Down-weighting overlapping genes improves gene set analysis\"<\/a>. <i>BMC Bioinformatics<\/i> <b>13<\/b>: 136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-13-136\" target=\"_blank\">10.1186\/1471-2105-13-136<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3443069\/\" target=\"_blank\">PMC3443069<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22713124\" target=\"_blank\">22713124<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3443069\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3443069<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Down-weighting+overlapping+genes+improves+gene+set+analysis&rft.jtitle=BMC+Bioinformatics&rft.aulast=Tarca%2C+A.L.%3B+Draghici%2C+S.%3B+Bhatti%2C+G.+et+al.&rft.au=Tarca%2C+A.L.%3B+Draghici%2C+S.%3B+Bhatti%2C+G.+et+al.&rft.date=2012&rft.volume=13&rft.pages=136&rft_id=info:doi\/10.1186%2F1471-2105-13-136&rft_id=info:pmc\/PMC3443069&rft_id=info:pmid\/22713124&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3443069&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TomfohrPath05-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TomfohrPath05_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tomfohr, J.; Lu, J.; Kepler, T.B. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1261155\" target=\"_blank\">\"Pathway level analysis of gene expression using singular value decomposition\"<\/a>. <i>BMC Bioinformatics<\/i> <b>6<\/b>: 225. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-6-225\" target=\"_blank\">10.1186\/1471-2105-6-225<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1261155\/\" target=\"_blank\">PMC1261155<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16156896\" target=\"_blank\">16156896<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1261155\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1261155<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pathway+level+analysis+of+gene+expression+using+singular+value+decomposition&rft.jtitle=BMC+Bioinformatics&rft.aulast=Tomfohr%2C+J.%3B+Lu%2C+J.%3B+Kepler%2C+T.B.&rft.au=Tomfohr%2C+J.%3B+Lu%2C+J.%3B+Kepler%2C+T.B.&rft.date=2005&rft.volume=6&rft.pages=225&rft_id=info:doi\/10.1186%2F1471-2105-6-225&rft_id=info:pmc\/PMC1261155&rft_id=info:pmid\/16156896&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1261155&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TarcaANovel09-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TarcaANovel09_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tarca, A.L.; Draghici, S.; Khatri, P. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2732297\" target=\"_blank\">\"A novel signaling pathway impact analysis\"<\/a>. <i>Bioinformatics<\/i> <b>25<\/b> (1): 75\u201382. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtn577\" target=\"_blank\">10.1093\/bioinformatics\/btn577<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2732297\/\" target=\"_blank\">PMC2732297<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18990722\" target=\"_blank\">18990722<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2732297\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2732297<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+novel+signaling+pathway+impact+analysis&rft.jtitle=Bioinformatics&rft.aulast=Tarca%2C+A.L.%3B+Draghici%2C+S.%3B+Khatri%2C+P.+et+al.&rft.au=Tarca%2C+A.L.%3B+Draghici%2C+S.%3B+Khatri%2C+P.+et+al.&rft.date=2009&rft.volume=25&rft.issue=1&rft.pages=75%E2%80%9382&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtn577&rft_id=info:pmc\/PMC2732297&rft_id=info:pmid\/18990722&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2732297&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NaeemRigor12-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NaeemRigor12_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-NaeemRigor12_21-1\">21.1<\/a><\/sup> <sup><a href=\"#cite_ref-NaeemRigor12_21-2\">21.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Naeem, H.; Zimmer, R.; Tavakkolkhah, P. et al. (2012). \"Rigorous assessment of gene set enrichment tests\". <i>Bioinformatics<\/i> <b>28<\/b> (11): 1480-6. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbts164\" target=\"_blank\">10.1093\/bioinformatics\/bts164<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22492315\" target=\"_blank\">22492315<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Rigorous+assessment+of+gene+set+enrichment+tests&rft.jtitle=Bioinformatics&rft.aulast=Naeem%2C+H.%3B+Zimmer%2C+R.%3B+Tavakkolkhah%2C+P.+et+al.&rft.au=Naeem%2C+H.%3B+Zimmer%2C+R.%3B+Tavakkolkhah%2C+P.+et+al.&rft.date=2012&rft.volume=28&rft.issue=11&rft.pages=1480-6&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbts164&rft_id=info:pmid\/22492315&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TarcaAComp13-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TarcaAComp13_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-TarcaAComp13_22-1\">22.1<\/a><\/sup> <sup><a href=\"#cite_ref-TarcaAComp13_22-2\">22.2<\/a><\/sup> <sup><a href=\"#cite_ref-TarcaAComp13_22-3\">22.3<\/a><\/sup> <sup><a href=\"#cite_ref-TarcaAComp13_22-4\">22.4<\/a><\/sup> <sup><a href=\"#cite_ref-TarcaAComp13_22-5\">22.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tarca, A.L.; Bhatti, G.; Romero, R. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829842\" target=\"_blank\">\"A comparison of gene set analysis methods in terms of sensitivity, prioritization and specificity\"<\/a>. <i>PLoS One<\/i> <b>8<\/b> (11): e79217. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0079217\" target=\"_blank\">10.1371\/journal.pone.0079217<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3829842\/\" target=\"_blank\">PMC3829842<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24260172\" target=\"_blank\">24260172<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829842\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829842<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comparison+of+gene+set+analysis+methods+in+terms+of+sensitivity%2C+prioritization+and+specificity&rft.jtitle=PLoS+One&rft.aulast=Tarca%2C+A.L.%3B+Bhatti%2C+G.%3B+Romero%2C+R.&rft.au=Tarca%2C+A.L.%3B+Bhatti%2C+G.%3B+Romero%2C+R.&rft.date=2013&rft.volume=8&rft.issue=11&rft.pages=e79217&rft_id=info:doi\/10.1371%2Fjournal.pone.0079217&rft_id=info:pmc\/PMC3829842&rft_id=info:pmid\/24260172&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3829842&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Bayerlov.C3.A1Comp15-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Bayerlov.C3.A1Comp15_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-Bayerlov.C3.A1Comp15_23-1\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-Bayerlov.C3.A1Comp15_23-2\">23.2<\/a><\/sup> <sup><a href=\"#cite_ref-Bayerlov.C3.A1Comp15_23-3\">23.3<\/a><\/sup> <sup><a href=\"#cite_ref-Bayerlov.C3.A1Comp15_23-4\">23.4<\/a><\/sup> <sup><a href=\"#cite_ref-Bayerlov.C3.A1Comp15_23-5\">23.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bayerlov\u00e1, M.; Jung, K.; Kramer, F. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4618947\" target=\"_blank\">\"Comparative study on gene set and pathway topology-based enrichment methods\"<\/a>. <i>BMC Bioinformatics<\/i> <b>16<\/b>: 334. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12859-015-0751-5\" target=\"_blank\">10.1186\/s12859-015-0751-5<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4618947\/\" target=\"_blank\">PMC4618947<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26489510\" target=\"_blank\">26489510<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4618947\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4618947<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comparative+study+on+gene+set+and+pathway+topology-based+enrichment+methods&rft.jtitle=BMC+Bioinformatics&rft.aulast=Bayerlov%C3%A1%2C+M.%3B+Jung%2C+K.%3B+Kramer%2C+F.+et+al.&rft.au=Bayerlov%C3%A1%2C+M.%3B+Jung%2C+K.%3B+Kramer%2C+F.+et+al.&rft.date=2015&rft.volume=16&rft.pages=334&rft_id=info:doi\/10.1186%2Fs12859-015-0751-5&rft_id=info:pmc\/PMC4618947&rft_id=info:pmid\/26489510&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4618947&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JaakkolaEmp16-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JaakkolaEmp16_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jaakkola, M.K.; Elo, L.L. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4793894\" target=\"_blank\">\"Empirical comparison of structure-based pathway methods\"<\/a>. <i>Briefings in Bioinformatics<\/i> <b>17<\/b> (2): 336\u201345. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbv049\" target=\"_blank\">10.1093\/bib\/bbv049<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4793894\/\" target=\"_blank\">PMC4793894<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26197809\" target=\"_blank\">26197809<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4793894\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4793894<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Empirical+comparison+of+structure-based+pathway+methods&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Jaakkola%2C+M.K.%3B+Elo%2C+L.L.&rft.au=Jaakkola%2C+M.K.%3B+Elo%2C+L.L.&rft.date=2016&rft.volume=17&rft.issue=2&rft.pages=336%E2%80%9345&rft_id=info:doi\/10.1093%2Fbib%2Fbbv049&rft_id=info:pmc\/PMC4793894&rft_id=info:pmid\/26197809&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4793894&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeMeyerAssess16-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DeMeyerAssess16_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-DeMeyerAssess16_25-1\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">De Meyer, S. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/libstore.ugent.be\/fulltxt\/RUG01\/002\/272\/484\/RUG01-002272484_2016_0001_AC.pdf\" target=\"_blank\">\"Assessing the performance of network crosstalk analysis combined with clustering\"<\/a> (PDF). Univeriteit Gent<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/libstore.ugent.be\/fulltxt\/RUG01\/002\/272\/484\/RUG01-002272484_2016_0001_AC.pdf\" target=\"_blank\">https:\/\/libstore.ugent.be\/fulltxt\/RUG01\/002\/272\/484\/RUG01-002272484_2016_0001_AC.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Assessing+the+performance+of+network+crosstalk+analysis+combined+with+clustering&rft.atitle=&rft.aulast=De+Meyer%2C+S.&rft.au=De+Meyer%2C+S.&rft.date=2016&rft.pub=Univeriteit+Gent&rft_id=https%3A%2F%2Flibstore.ugent.be%2Ffulltxt%2FRUG01%2F002%2F272%2F484%2FRUG01-002272484_2016_0001_AC.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LimComp20-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LimComp20_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lim, S.; Lee, S.; Jung, I. et al. (2020). \"Comprehensive and critical evaluation of individualized pathway activity measurement tools on pan-cancer data\". <i>Briefings in Bioinformatics<\/i> <b>21<\/b> (1): 36\u201346. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbby125\" target=\"_blank\">10.1093\/bib\/bby125<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30462155\" target=\"_blank\">30462155<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comprehensive+and+critical+evaluation+of+individualized+pathway+activity+measurement+tools+on+pan-cancer+data&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Lim%2C+S.%3B+Lee%2C+S.%3B+Jung%2C+I.+et+al.&rft.au=Lim%2C+S.%3B+Lee%2C+S.%3B+Jung%2C+I.+et+al.&rft.date=2020&rft.volume=21&rft.issue=1&rft.pages=36%E2%80%9346&rft_id=info:doi\/10.1093%2Fbib%2Fbby125&rft_id=info:pmid\/30462155&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NguyenIndent19-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NguyenIndent19_27-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-NguyenIndent19_27-1\">27.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nguyen, T.-M.; Shafi, A.; Nguyen, T. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6784345\" target=\"_blank\">\"Identifying significantly impacted pathways: A comprehensive review and assessment\"<\/a>. <i>Genome Biology<\/i> <b>20<\/b> (1): 203. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13059-019-1790-4\" target=\"_blank\">10.1186\/s13059-019-1790-4<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6784345\/\" target=\"_blank\">PMC6784345<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31597578\" target=\"_blank\">31597578<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6784345\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6784345<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identifying+significantly+impacted+pathways%3A+A+comprehensive+review+and+assessment&rft.jtitle=Genome+Biology&rft.aulast=Nguyen%2C+T.-M.%3B+Shafi%2C+A.%3B+Nguyen%2C+T.+et+al.&rft.au=Nguyen%2C+T.-M.%3B+Shafi%2C+A.%3B+Nguyen%2C+T.+et+al.&rft.date=2019&rft.volume=20&rft.issue=1&rft.pages=203&rft_id=info:doi\/10.1186%2Fs13059-019-1790-4&rft_id=info:pmc\/PMC6784345&rft_id=info:pmid\/31597578&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6784345&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaAComp19-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaAComp19_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ma, J.; Shojaie, A.; Michailidis, G. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6829999\" target=\"_blank\">\"A comparative study of topology-based pathway enrichment analysis methods\"<\/a>. <i>BMC Bioinformatics<\/i> <b>20<\/b> (1): 546. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12859-019-3146-1\" target=\"_blank\">10.1186\/s12859-019-3146-1<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6829999\/\" target=\"_blank\">PMC6829999<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31684881\" target=\"_blank\">31684881<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6829999\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6829999<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comparative+study+of+topology-based+pathway+enrichment+analysis+methods&rft.jtitle=BMC+Bioinformatics&rft.aulast=Ma%2C+J.%3B+Shojaie%2C+A.%3B+Michailidis%2C+G.&rft.au=Ma%2C+J.%3B+Shojaie%2C+A.%3B+Michailidis%2C+G.&rft.date=2019&rft.volume=20&rft.issue=1&rft.pages=546&rft_id=info:doi\/10.1186%2Fs12859-019-3146-1&rft_id=info:pmc\/PMC6829999&rft_id=info:pmid\/31684881&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6829999&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZylaGene19-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ZylaGene19_29-0\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-ZylaGene19_29-1\">29.1<\/a><\/sup> <sup><a href=\"#cite_ref-ZylaGene19_29-2\">29.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zyla, J.; Marczyk, M.; Doaszewska, T. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6954644\" target=\"_blank\">\"Gene set enrichment for reproducible science: Comparison of CERNO and eight other algorithms\"<\/a>. <i>Bioinformatics<\/i> <b>35<\/b> (24): 5146\u201354. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtz447\" target=\"_blank\">10.1093\/bioinformatics\/btz447<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6954644\/\" target=\"_blank\">PMC6954644<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31165139\" target=\"_blank\">31165139<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6954644\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6954644<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+set+enrichment+for+reproducible+science%3A+Comparison+of+CERNO+and+eight+other+algorithms&rft.jtitle=Bioinformatics&rft.aulast=Zyla%2C+J.%3B+Marczyk%2C+M.%3B+Doaszewska%2C+T.+et+al.&rft.au=Zyla%2C+J.%3B+Marczyk%2C+M.%3B+Doaszewska%2C+T.+et+al.&rft.date=2019&rft.volume=35&rft.issue=24&rft.pages=5146%E2%80%9354&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtz447&rft_id=info:pmc\/PMC6954644&rft_id=info:pmid\/31165139&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6954644&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Geistlinger-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Geistlinger_30-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-Geistlinger_30-1\">30.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Geistlinger, L.; Csaba, G.; Santarelli, M. et al. (2021). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7820859\" target=\"_blank\">\"Toward a gold standard for benchmarking gene set enrichment analysis\"<\/a>. <i>Briefings in Bioinformatics<\/i> <b>22<\/b> (1): 545\u201356. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbz158\" target=\"_blank\">10.1093\/bib\/bbz158<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7820859\/\" target=\"_blank\">PMC7820859<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32026945\" target=\"_blank\">32026945<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7820859\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7820859<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+gold+standard+for+benchmarking+gene+set+enrichment+analysis&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Geistlinger%2C+L.%3B+Csaba%2C+G.%3B+Santarelli%2C+M.+et+al.&rft.au=Geistlinger%2C+L.%3B+Csaba%2C+G.%3B+Santarelli%2C+M.+et+al.&rft.date=2021&rft.volume=22&rft.issue=1&rft.pages=545%E2%80%9356&rft_id=info:doi\/10.1093%2Fbib%2Fbbz158&rft_id=info:pmc\/PMC7820859&rft_id=info:pmid\/32026945&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7820859&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GlaabEnrich12-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GlaabEnrich12_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Glaab, E.; Baudot, A.; Krasnogor, N. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3436816\" target=\"_blank\">\"EnrichNet: Network-based gene set enrichment analysis\"<\/a>. <i>Briefings in Bioinformatics<\/i> <b>28<\/b> (18): i451-i457. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbts389\" target=\"_blank\">10.1093\/bioinformatics\/bts389<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3436816\/\" target=\"_blank\">PMC3436816<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22962466\" target=\"_blank\">22962466<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3436816\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3436816<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EnrichNet%3A+Network-based+gene+set+enrichment+analysis&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Glaab%2C+E.%3B+Baudot%2C+A.%3B+Krasnogor%2C+N.+et+al.&rft.au=Glaab%2C+E.%3B+Baudot%2C+A.%3B+Krasnogor%2C+N.+et+al.&rft.date=2012&rft.volume=28&rft.issue=18&rft.pages=i451-i457&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbts389&rft_id=info:pmc\/PMC3436816&rft_id=info:pmid\/22962466&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3436816&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DongLEGO16-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DongLEGO16_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dong, X.; Hao, Y.; Wang, X. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4707541\" target=\"_blank\">\"LEGO: A novel method for gene set over-representation analysis by incorporating network-based gene weights\"<\/a>. <i>Scientific Reports<\/i> <b>6<\/b>: 18871. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fsrep18871\" target=\"_blank\">10.1038\/srep18871<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4707541\/\" target=\"_blank\">PMC4707541<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26750448\" target=\"_blank\">26750448<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4707541\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4707541<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LEGO%3A+A+novel+method+for+gene+set+over-representation+analysis+by+incorporating+network-based+gene+weights&rft.jtitle=Scientific+Reports&rft.aulast=Dong%2C+X.%3B+Hao%2C+Y.%3B+Wang%2C+X.+et+al.&rft.au=Dong%2C+X.%3B+Hao%2C+Y.%3B+Wang%2C+X.+et+al.&rft.date=2016&rft.volume=6&rft.pages=18871&rft_id=info:doi\/10.1038%2Fsrep18871&rft_id=info:pmc\/PMC4707541&rft_id=info:pmid\/26750448&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4707541&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlhamdooshEasy17-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlhamdooshEasy17_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alhamdoosh, M.; Law, C.W.; Tian, L. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5747338\" target=\"_blank\">\"Easy and efficient ensemble gene set testing with EGSEA\"<\/a>. <i>F1000Research<\/i> <b>6<\/b>: 2010. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.12688%2Ff1000research.12544.1\" target=\"_blank\">10.12688\/f1000research.12544.1<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5747338\/\" target=\"_blank\">PMC5747338<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29333246\" target=\"_blank\">29333246<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5747338\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5747338<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Easy+and+efficient+ensemble+gene+set+testing+with+EGSEA&rft.jtitle=F1000Research&rft.aulast=Alhamdoosh%2C+M.%3B+Law%2C+C.W.%3B+Tian%2C+L.+et+al.&rft.au=Alhamdoosh%2C+M.%3B+Law%2C+C.W.%3B+Tian%2C+L.+et+al.&rft.date=2017&rft.volume=6&rft.pages=2010&rft_id=info:doi\/10.12688%2Ff1000research.12544.1&rft_id=info:pmc\/PMC5747338&rft_id=info:pmid\/29333246&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5747338&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-34\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Kokemuller, N. (28 January 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/smallbusiness.chron.com\/people-buy-brand-names-69654.html\" target=\"_blank\">\"Why Do People Buy Brand Names?\"<\/a>. <i>Chron<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/smallbusiness.chron.com\/people-buy-brand-names-69654.html\" target=\"_blank\">https:\/\/smallbusiness.chron.com\/people-buy-brand-names-69654.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Why+Do+People+Buy+Brand+Names%3F&rft.atitle=Chron&rft.aulast=Kokemuller%2C+N.&rft.au=Kokemuller%2C+N.&rft.date=28+January+2019&rft_id=https%3A%2F%2Fsmallbusiness.chron.com%2Fpeople-buy-brand-names-69654.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Barab.C3.A1siTheForm18-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Barab.C3.A1siTheForm18_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Barab\u00e1si, A.-L. (2018). <i>The Formula: The Universal Laws of Success<\/i>. Little, Brown and Company. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780316505499.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Formula%3A+The+Universal+Laws+of+Success&rft.aulast=Barab%C3%A1si%2C+A.-L.&rft.au=Barab%C3%A1si%2C+A.-L.&rft.date=2018&rft.pub=Little%2C+Brown+and+Company&rft.isbn=9780316505499&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeePrior16-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeePrior16_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lee, J.; Jo, K.; Lee, S. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5259824\" target=\"_blank\">\"Prioritizing biological pathways by recognizing context in time-series gene expression data\"<\/a>. <i>BMC Bioinformatics<\/i> <b>17<\/b> (Suppl 17): 477. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12859-016-1335-8\" target=\"_blank\">10.1186\/s12859-016-1335-8<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5259824\/\" target=\"_blank\">PMC5259824<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28155707\" target=\"_blank\">28155707<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5259824\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5259824<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Prioritizing+biological+pathways+by+recognizing+context+in+time-series+gene+expression+data&rft.jtitle=BMC+Bioinformatics&rft.aulast=Lee%2C+J.%3B+Jo%2C+K.%3B+Lee%2C+S.+et+al.&rft.au=Lee%2C+J.%3B+Jo%2C+K.%3B+Lee%2C+S.+et+al.&rft.date=2016&rft.volume=17&rft.issue=Suppl+17&rft.pages=477&rft_id=info:doi\/10.1186%2Fs12859-016-1335-8&rft_id=info:pmc\/PMC5259824&rft_id=info:pmid\/28155707&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5259824&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VitekBench20-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VitekBench20_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Vitek, O.; Robinson, M.D., ed. (2020). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.biomedcentral.com\/collections\/benchmarkingstudies\" target=\"_blank\">\"Benchmarking Studies\"<\/a>. <i>Genome Biology<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.biomedcentral.com\/collections\/benchmarkingstudies\" target=\"_blank\">https:\/\/www.biomedcentral.com\/collections\/benchmarkingstudies<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Benchmarking+Studies&rft.atitle=Genome+Biology&rft.date=2020&rft_id=https%3A%2F%2Fwww.biomedcentral.com%2Fcollections%2Fbenchmarkingstudies&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PLoSComp18-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PLoSComp18_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/collections.plos.org\/collection\/benchmarking\/\" target=\"_blank\">\"PLOS Computational Biology Benchmarking Collection\"<\/a>. PLoS. 29 November 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/collections.plos.org\/collection\/benchmarking\/\" target=\"_blank\">https:\/\/collections.plos.org\/collection\/benchmarking\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=PLOS+Computational+Biology+Benchmarking+Collection&rft.atitle=&rft.date=29+November+2018&rft.pub=PLoS&rft_id=https%3A%2F%2Fcollections.plos.org%2Fcollection%2Fbenchmarking%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EfroniIdent07-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EfroniIdent07_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Efroni, S.; Schaefer, C.F.; Buetow, K.H. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1855990\" target=\"_blank\">\"Identification of key processes underlying cancer phenotypes using biologic pathway analysis\"<\/a>. <i>PLoS One<\/i> <b>2<\/b> (5): e425. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0000425\" target=\"_blank\">10.1371\/journal.pone.0000425<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1855990\/\" target=\"_blank\">PMC1855990<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17487280\" target=\"_blank\">17487280<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1855990\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1855990<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identification+of+key+processes+underlying+cancer+phenotypes+using+biologic+pathway+analysis&rft.jtitle=PLoS+One&rft.aulast=Efroni%2C+S.%3B+Schaefer%2C+C.F.%3B+Buetow%2C+K.H.&rft.au=Efroni%2C+S.%3B+Schaefer%2C+C.F.%3B+Buetow%2C+K.H.&rft.date=2007&rft.volume=2&rft.issue=5&rft.pages=e425&rft_id=info:doi\/10.1371%2Fjournal.pone.0000425&rft_id=info:pmc\/PMC1855990&rft_id=info:pmid\/17487280&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1855990&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WacholderValid93-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WacholderValid93_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wacholder, S.; Armstrong, B.; Hartge, P. (1993). \"Validation studies using an alloyed gold standard\". <i>American Journal of Epidemiology<\/i> <b>137<\/b> (11): 1251\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Foxfordjournals.aje.a116627\" target=\"_blank\">10.1093\/oxfordjournals.aje.a116627<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/8322765\" target=\"_blank\">8322765<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Validation+studies+using+an+alloyed+gold+standard&rft.jtitle=American+Journal+of+Epidemiology&rft.aulast=Wacholder%2C+S.%3B+Armstrong%2C+B.%3B+Hartge%2C+P.&rft.au=Wacholder%2C+S.%3B+Armstrong%2C+B.%3B+Hartge%2C+P.&rft.date=1993&rft.volume=137&rft.issue=11&rft.pages=1251%E2%80%938&rft_id=info:doi\/10.1093%2Foxfordjournals.aje.a116627&rft_id=info:pmid\/8322765&rfr_id=info:sid\/en.wikipedia.org:Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150558\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.793 seconds\nReal time usage: 1.514 seconds\nPreprocessor visited node count: 33809\/1000000\nPreprocessor generated node count: 40699\/1000000\nPost\u2010expand include size: 311988\/2097152 bytes\nTemplate argument size: 98705\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 701.366 1 - -total\n 87.88% 616.363 1 - Template:Reflist\n 74.77% 524.441 40 - Template:Citation\/core\n 68.91% 483.339 34 - Template:Cite_journal\n 11.74% 82.322 97 - Template:Citation\/identifier\n 7.34% 51.492 5 - Template:Cite_web\n 5.45% 38.254 1 - Template:Infobox_journal_article\n 5.19% 36.404 1 - Template:Infobox\n 4.74% 33.219 43 - Template:Citation\/make_link\n 4.36% 30.565 222 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12464-0!*!0!!en!5!* and timestamp 20210705150557 and revision id 42308\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis\">https:\/\/www.limswiki.org\/index.php\/Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","7ef870605af2fae6ca37ea1b35850e50_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/94\/Fig1_Xie_BMCBioinfo21_22.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/39\/Fig2_Xie_BMCBioinfo21_22.png"],"7ef870605af2fae6ca37ea1b35850e50_timestamp":1625497557,"cd03d5b8e3a613ce4ec49dfb1dc3ff59_type":"article","cd03d5b8e3a613ce4ec49dfb1dc3ff59_title":"Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results (Jin et al. 2021)","cd03d5b8e3a613ce4ec49dfb1dc3ff59_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results","cd03d5b8e3a613ce4ec49dfb1dc3ff59_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDevelopment and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test resultsJournal\n \nBMC Medical Informatics and Decision MakingAuthor(s)\n \nJin, Di; Wang, Dezhi; Wang, Jiajia; Li, Bijuan; Cheng, Yating; Mo, Nanxun; Deng, Xiaoyan; Tao, RanAuthor affiliation(s)\n \nJinan Kingmed Center for Clinical Laboratory, Guangzhou Medical UniversityPrimary contact\n \nEmail: Online formYear published\n \n2021Volume and issue\n \n21Article #\n \n174DOI\n \n10.1186\/s12911-021-01545-3ISSN\n \n1472-6947Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-021-01545-3Download\n \nhttps:\/\/bmcmedinformdecismak.biomedcentral.com\/track\/pdf\/10.1186\/s12911-021-01545-3.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n\n2.1 Current status and challenges \n\n\n3 Methods \n\n3.1 System design \n3.2 Validation scheme \n3.3 Correctness verification \n3.4 Integrety validation \n3.5 Accuracy guarantee \n3.6 Data collection \n3.7 Time consumption statistics \n3.8 Satisfaction survey \n\n\n4 Results \n\n4.1 Correctness verification results \n4.2 Integrity validation results \n4.3 Comparison of the two methods \n4.4 Satisfaction survey \n\n\n5 Discussion \n6 Conclusions \n7 Abbreviations \n8 Acknowledgements \n\n8.1 Contributions \n8.2 Funding \n8.3 Ethics approval and consent to participate \n8.4 Availability of data and materials \n8.5 Competing interests \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nBackground: For laboratory informatics applications, validation of the autoverification function is one of the critical steps to confirm its effectiveness before use. It is crucial to verify whether the programmed algorithm follows the expected logic and produces the expected results. This process has always relied on the assessment of human\u2013machine consistency and is mostly a manually recorded and time-consuming activity with inherent subjectivity and arbitrariness that cannot guarantee a comprehensive, timely, and continuous effectiveness evaluation of the autoverification function. To overcome these inherent limitations, we independently developed and implemented a laboratory information system (LIS)-based validation system for autoverification.\nMethods: We developed a correctness verification and integrity validation method (hereinafter referred to as the \"new method\") in the form of a human\u2013machine dialog. The system records personnel review steps and determines whether the human\u2013machine review results are consistent. Laboratory personnel then analyze the reasons for any inconsistency according to system prompts, add to or modify rules, reverify, and finally improve the accuracy of autoverification.\nResults: The validation system was successfully established and implemented. For a dataset consisting of 833 rules for 30 assays, 782 rules (93.87%) were successfully verified in the correctness verification phase, and 51 rules were deleted due to execution errors. In the integrity validation phase, 24 projects were easily verified, while the other six projects still required the additional rules or changes to the rule settings. Taking the Hepatitis B virus test as an example, from the setting of 65 rules to the automated releasing of 3,000 reports, the validation time was reduced from 452 (manual verification) to 275 hours (new method), a reduction in validation time of 177 hours. Furthermore, 94.6% (168\/182) of laboratory users believed the new method greatly reduced the workload, effectively controlled the report risk, and felt satisfied. Since 2019, over 3.5 million reports have been automatically reviewed and issued without a single clinical complaint.\nConclusion: To the best of our knowledge, this is the first report to realize autoverification validation as a human\u2013machine interaction. The new method effectively controls the risks of autoverification, shortens time consumption, and improves the efficiency of laboratory verification.\nKeywords: autoverification, correctness verification, integrity validation, human\u2013computer interaction, risk management, laboratory information system\n\nBackground \nAutoverification\u2014the use of automated computer-based rules to initially validate laboratory test results[1]\u2014is a powerful tool for the batch processing of test results and has been widely used in recent years. It has obvious advantages in reducing reporting errors, shortening turnaround time (TAT), and improving audit efficiency.[1][2][3][4][5]\n\nCurrent status and challenges \nOur self-developed autoverification system has been used for six years in many disciplines, such as biochemistry, immunology, hematology, microbiology, molecular diagnostics, and pathology. To date, 25,487 rules have been set. The system judges test results 1.1 million times a day and provides audit recommendations for 250,000 report forms, accounting for 87% of the total number of report forms. Approximately 80,000 reports are automatically generated every day. To ensure the effectiveness and safety of the autoverification system, its validation process is very important. The College of American Pathologists' laboratory accreditation checklist item GEN.43875[6] and International Organization for Standardization's ISO 15189:2012 requirement 5.9.2b[7] both require that autoverification systems undergo functional verification before use.\nAccording to published studies, in laboratories that use autoverification, the majority of laboratories have performed personnel-based and automatic system audits with the same results\u2014manually recorded consistency\u2014and reached a conclusion after a statistical analysis of the results.[2][4][8][9] The manual verification method is less difficult to operate but has the following limitations:\n\r\n\n\n Massive validation workload: Based on the requirements of WS\/T 616-2018 (a China Health Organization recommended standard)[10] for validation of the autoverification of quantitative clinical laboratory test results, every test and every sample type involved in the autoverification procedure should be tested; the validation time should be no less than three months and\/or the number of reports released should be no less than 50,000; and periodic verification should be performed every year for no less than 10 working days and\/or for no less than 5,000 reports. The validation workload is large, and it is difficult to rely on manual comparison and recording, which greatly increases the post-analytical workload.\n Reporting risk: During manual verification, personnel are prone to inertia or judgment errors. The lack of a system control mechanism for this kind of validation can generate reporting risks and directly affect clinical diagnosis and treatment.[2]\nTherefore, there is an urgent need to design a verification method that minimizes the workload and systematically controls risks. We report a rule verification system with a small workload and ease of operation that can be used as a reference for self-built and automatic test auditing for laboratories and manufacturers.\n\nMethods \nSystem design \nBased on the Clinical and Laboratory Standards Institute's (CLSI) AUTO10 standard[11] and current review processes, we established an autoverification system including 11 rule categories. Technicians set the rules according to audit requirements and rule categories. Each item can set multiple rules, including limited range check, combined mode judgment, Delta check, sampling time validity judgment, sample abnormality judgment (e.g., hemolysis, lipemia), and quality control check. The autoverification system determines whether the report is abnormal according to the rules. Tests that do not trigger contradiction mode are displayed in green, while failed tests (triggering rules, contradictory modes set by the rules) are displayed in red, and the cause of the contradiction is indicated. If all the tests in the report are green, the barcode of the report is also green. If any test in the report is red, the report shows a red barcode, which signals a warning in the system.\nAccording to the above steps, the autoverification system displays colors and abnormal prompts after judging the rules in a process called automatic early warning. The automatic warning is only for judgment and is not involved in the decision to issue a report. Based on this, the system automatically sends out a report with a green barcode in a process called automated reporting. Automatic early warning and automatic reporting comprise autoverification. This system is especially useful in the review of complex diagnostic projects (e.g., molecular diagnostics, pathological testing). These projects prompt absurd values from personnel. For some moderately complex projects (e.g., biochemical, blood), the combination of report reviewing, automatic warning, and automated reporting is equivalent to the autoverification system in a large number of literature reports and laboratory information system (LIS) automatic reports. The autoverification process used by our laboratory is shown in Fig. 1.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 1. The autoverification process. Single test results must meet all the warning rules at the same time. The autoverification algorithm can identify those samples requiring manual review that do not meet the laboratory\u2019s criteria for autoverification. If the automated reporting switch is not activated, then reports that pass the automatic warning step are manually issued. If the automated reporting switch is turned on and all tests on the report pass their warning rules, then the system automatically releases the report.\n\n\n\nValidation scheme \nOn the premise that automatic audits are divided into automatic warnings and automatic reports, we divide the verification system into two stages. The first stage is called correctness verification, which verifies that the operation of the rules is consistent with the expectations set by the personnel. If there is a problem, the responsible party may be the program development department. The second stage is called integrity validation. Based on the results from the first stage, this stage verifies whether the set rules include all the elements from the personnel\u2019s audit report. The functional design of the two-stage system is shown in Table 1.\n\n\n\n\n\n\n\nTable 1. Two validation methods designed for two parts of the autoverification system\n\n\nPhase\n\nObject\n\nValidation method\n\nExplanation\n\nInconsistent solutions\n\n\nAutomatic warning\n\nWarning rules\n\nCorrectness verification\n\nTo verify that the warning rules behave as expected and produce the expected outcome\n\nIf the warning rule setting is wrong, delete and reset the rules\n\n\nAutomated reporting\n\nLaboratory tests\n\nIntegrity validation\n\nTo confirm that the laboratory test results that pass the automatic warning can be reported automatically\n\nAdd more warning rules according to the laboratory report criteria\n\n\n\nCorrectness verification \nThe correctness verification phase confirms whether the execution of a single rule is correct. It is implemented as follows: (1) For newly added rules, the system adds the label \"Pending Verification.\" (2) When the report is reviewed, the system displays the rule judgment result, and a purple color block is displayed to remind the staff to judge whether the execution result of the \"Pending Verification\" rule is correct. (3) The staff input the judgment result. (4) The system changes the rule status according to the staff input. If it is consistent, the rule label is set to \"verified,\" prompting the personnel to continue to the next stage of verification. If it is inconsistent, the staff is prompted to delete the rule. Figure 2 is a diagram representing this correctness verification process using the example of C-reactive protein (CRP). Figure 3 shows an example of the correctness verification interface.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 2. Schematic diagram of the correctness verification using the example of C-reactive protein (CRP). The CRP test result was 1.8 mg\/l and passed quality control. The autoverification system searched all the rules for the CRP and hit two of them, No. 001879 and No. 002009. The No. 001879 rule (verified) checks whether the CRP result has passed the quality control. The No. 002009 rule (pending verification) intercepts the results greater than or equal to 5. Therefore, when No. 002009 is triggered, the warning information of the sample appears purple, indicating that the technician needs to confirm whether the warning result is consistent with the manual judgment. In the correctness verification interface (as shown in the subsequent Fig. 3), the system provides two options, the human\u2013machine judgment is consistent or the system judges incorrectly. The technician can confirm that the rule is performing correctly and change its status to \u201cverified.\u201d\n\n\n\n\n\n\n\n\n\n\n\n\nFigure 3. Correctness verification interface. The result of CRP passes automatic warning according to the No.002009 rule and displays green. The technician judges whether the automated warning operates correctly.\n\n\n\nIntegrety validation \nIntegrity validation can be started only after the correctness verification of all rules of a project is completed. It is implemented as follows. (1) After the report shows the result of the automatic warning, if the system detects that the report has been changed, a dialog box will pop up and ask the reviewer to select the reason for the modification. These reasons include (a) a rule execution error, (b) a rule setting value that is inappropriate, (c) the required addition of new rules, (d) the lack of involvement of other issues related to automatic review, and (e) automatic warning and prompt modification. The LIS records the modified content and the reasons for personnel analysis. (2) If the laboratory wants to implement automated reporting, a validation number, such as 5,000, can be set according to the complexity of the project review. (3) If the automatic warning result of the report is green (approved), the personnel will issue the report directly, and the validation number of the report will automatically increase by one. (4) If the validation number of all items on the report exceeds the set number, the report will be automatically released. (5) If the automatic warning result of the report is green (approved), but the result is modified, with the reason for the modification specified as any of a, b, or c, then the LIS will clear the validation number for the related items and stop automated reporting. Figure 4 shows the integrity validation process. The validation goals and validation amount for six projects are shown in Fig. 5.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 4. The integrity validation process.\n\n\n\n\n\n\n\n\n\n\n\n\nFigure 5. Integrity validation target number settings and recording interface. The validation targets of the six projects in the above figure are all 3000, and the validation number is between 1900 and 2500. The corresponding reports cannot be released automatically.\n\n\n\nAccuracy guarantee \nThe accuracy of the autoverification includes whether the rules can be identified and whether there are omissions (completeness) in the report review. Therefore, our method confirms the accuracy of the new method from these two aspects. We perform function correction and system improvement through correctness verification and integrity validation. In the validation system, we design the following logic to ensure the accuracy of the function:\n\n The new rule is automatically deleted if it fails the correctness verification within 10 days.\n The rule is not allowed to be modified.\n If the rule fails the correctness verification, it is forbidden to be converted.\n If the autoverification of a single project fails the integrity validation, the historical validation amount is cleared.\nData collection \nThe validation data of 30 assays from October 2019 to January 2020 were collected for analysis, and in total, 833 early warning rules were obtained. A total of 926,195 reports was used to evaluate the accuracy of the new method.\n\nTime consumption statistics \nWe used HBV as an example to introduce the comparison of the validation time before and after the new method was used. In the measurement of the validation time, we divided the complete autoverification into 10 stages. Time statistics were collected for manual verification and new method verification for each step. We used systematic records and estimates to develop time statistics for different stages.\n\nSatisfaction survey \nWe used questionnaires to evaluate the effectiveness of new methods used by laboratory technicians. The survey was launched using the online tool WJX, which feeds back the percentage of responses and the total numbers.\n\nResults \nCorrectness verification results \nAmong the 833 rules, 782 (93.88%) were successfully verified for correctness, with a total of 3,814 validations, including 2,230 (58.47%) released tests and 1,584 (41.53%) intercepted tests. The inconsistencies were verified, and 51 (0.06%) error rules were deleted. The reasons for verification failure are shown in Table 2.\n\n\n\n\n\n\n\nTable 2. List of reasons for correctness verification failure\n\n\nError type\n\nProportion (%)\n\nSample\n\nSolution\n\n\nHuman error\n\n63.3\n\nIncorrect English letter case in the text of the rules, resulting in no warning\n\nReset the rules\n\n\nSpecific warning target\n\n24.9\n\nEarly warning of diagnostic results and microscopy results in a special report interface for pathology\n\nAdd a supplementary algorithm code\n\n\nAlgorithm code error\n\n8.4\n\nHPV typing results could not be verified with the Delta Check; the results of the microbial project identification could not be correlated with a variety of drug sensitivity combinations\n\nFix the algorithm code\n\n\nSoftware compatibility problem\n\n3.4\n\nProblem with the precision of the number comparison script\n\nFix the algorithm code\n\n\n\nIntegrity validation results \nWe collected integrity validation data through system export and department feedback. The reasons provided for rule modification were automatic warning prompt and rule modification (5, 10.6%), rule execution error (0, 0%), improper setting values (15, 31.9%), new rule added (18, 38.3%), and no automatic warning involving other questions (9, 19.2%). The integrity of all projects was verified within one month, and the problems found are shown in Table 3.\n\n\n\n\n\n\n\nTable 3. List of reasons why integrity validation failed\n\n\nTest\n\nReason for not passing\n\nSolution\n\n\nHPV genotyping\n\nThere was no comprehensive analysis of the combined thin-layer cytology results\n\nAnalyze the results associated with thin-layer cytology\n\n\nUrea\n\nThe limit range was too wide\n\nReduce the limit range\n\n\nAlbumin\n\nReview of the detection system produces an error\n\nSpecify the detection system\n\n\nCBC\n\nTest results were checked only on the same day as the barcode\n\nExtend the backdating of the historical results\n\n\nHBsAg HBsAb HBeAg HBeAb HBcAb\n\nNot all composite mode scenarios were covered\n\nAdd a joint audit of the portfolio project results\n\n\nCortisol\n\nThere was no warning of abnormal rhythms\n\nAdd a rule about checking sampling time\n\n\n\nComparison of the two methods \nThe comparison of manual record analysis and the new method for different steps is shown in Table 4. The new method performs four automation steps, reduces the personnel workload, and automatically controls the enabling and disabling of automatic report release through system monitoring report modification. The increased accuracy verification can quickly eliminate rule setting exceptions and development loopholes while reducing the time needed for personnel analysis. The manual record analysis and the new method took 452 hours and 275 hours to complete, respectively.\n\n\n\n\n\n\n\nTable 4. Comparison of the time consumption (hours) of the two methods for verifying HBV reports for 3,000 cases. In the measurement of the validation time, we divided the complete autoverification into 10 stages. In steps 4\u20136, in total, 3,000 reports are used for statistics. The time consumption of the consistent work content in the new and old methods is subject to the following: the manual timing of the old method, such as steps 1, 2, 4, and 9; the inconsistent steps in the two methods; the new steps that are recorded in the system, such as step 3; the saving step time clearing, such as steps 5, 6, 7, and 10; and the remaining steps that are estimated, such as step 8. For automatic implementation, the time is calculated as zero. aReasons for invalid locking rules. bReduced workload. cControlled risks.\n\n\nSteps\n\nManual validation (h)\n\nNew method (h)\n\n\n1. Set 65 rules\n\n1.5\n\n1.5\n\n\n2. Perform Rule 130 test\n\n2.5\n\n2.5\n\n\n3. Correctness verification\n\n0\n\n0.25a\n\n\n4. Personnel comparison report and results review\n\n240\n\n240\n\n\n5. Record comparison result\n\n100\n\n0b\n\n\n6. Analysis of the verification number\n\n10\n\n0b\n\n\n7. Determine whether to activate automatic approval\n\n5\n\n0b\n\n\n8. Personnel analysis of the reasons for inconsistent audit results\n\n90\n\n30\n\n\n9. Add and modify rules\n\n1\n\n1\n\n\n10. Determine whether to turn off autoverification\n\n1\n\n0c\n\n\nTotal\n\n452\n\n275\n\n\n\nSatisfaction survey \nAfter using the new method for one year, we conducted a satisfaction survey of laboratory personnel who used the function. We distributed 182 questionnaires and recovered 168 of them, with a response rate of 92.3%. The survey results showed that 94.6% of laboratory users believed that the new method could greatly reduce the workload, effectively control the report risk, and produce satisfactory or very satisfactory assessments of the new method.\n\nDiscussion \nThe core of the use of autoverification lies in the validation of system functions and rules. Due to the complexity of these rules, it is impossible to find all the functional defects by relying solely on function validation before the system goes online, and even human input errors cannot be carried out in the validation.[12] Such functional defects must be found in actual application scenarios with multiple different rule settings, such as the incorrect input of full-width symbols, that is, correctness verification. Furthermore, the premise of rule verification is to include a review of the logic of all reviewers in the system, which can be discovered only in actual application scenarios. Additionally, integrity validation can be performed in actual application scenarios to truly find problems.[13][14]\nWe initially designed the system in two parts\u2014automatic warning and automated reporting\u2014to allow complex detection items (molecular and pathological examinations, final human reports, and system prompt errors) to be included in the automatic review. Laboratory technicians could then choose to address the needs of different measurements. These two parts correspond to two verification steps: the automatic warning portion that performs correctness verification, and the automated reporting portion that performs integrity validation.\nCompared with other systems reported in the literature[4], the advantages of the new method are mainly simplifying the verification process, reducing the verification workload, and ensuring the accuracy of the verification results. As shown in Fig. 6, in the manual verification scheme, the junior staff will review the results, and then the intermediate staff will repeat reviewing the results and combine the autoverification results to determine the human\u2013machine consistency to complete the autoverification function validation. The whole process focuses on human leadership. The new method uses system monitoring to judge the accuracy of the autoverification based on the operational trajectories of different personnel. By interacting with personnel, the system collects validation data and controls the operation of autoverification.\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 6. Schematic diagram of the process comparison between the manual method and the new method\n\n\n\nWe delineate the advantages of the new method, compared to manual validation, in Table 5.\n\n\n\n\n\n\n\nTable 5. Comparison of the advantages of the new method and manual verification\n\n\nAdvantages\n\nDifference\n\nManual validation\n\nNew method\n\nExplanation\n\n\nEfficiency improvement\n\nWhether to add extra workload?\n\nYES\n\nNO\n\nNo additional personnel are required to manually record the reason for the inconsistency. The new method is that the system completes judgment and records while personnel review the reports normally. The system will control the operation of the autoverification program based on the consistency results.\n\n\nCan the cause of inconsistency be quickly determined?\n\nNO\n\nYES\n\nThe main reasons for the inconsistency are abnormal rule settings and lack of necessary rules. The new method correspondingly sets up correctness verification and integrity validation for these two main reasons. In different verification stages, only the main reason for that stage can be traced back.\n\n\nRisk control\n\nIs it possible to skip the validation process?\n\nYES\n\nNO\n\nStarting from setting the rules, the system will pull the validation process, and no validation link can be skipped.\n\n\nWhether to ensure sufficient amount of validation data?\n\nNO\n\nYES\n\nIn the process of normal personnel issuance, the system will truthfully record the validation data. Before the set data volume is reached, the automated reporting function is prohibited.\n\n\nCan autoverification be used in the case of failed validation?\n\nYES\n\nNO\n\nWhen the system confirms that the validation fails due to a defect in the autoverification, it will prohibit the rule conversion or the automated reporting from being enabled.\n\n\n\nCompared with the traditional method, the true positives and false positives of the \"personal and machine-based audit results\" are easy to understand, but if the indicators are abnormal, it can be difficult to find the cause of this abnormality, especially after all the reasons are verified after thousands of reports are released.[15] Consequently, the audit scenario has become blurred in the auditor's memory, and it becomes inefficient to check the problems one by one. The process-based validation scheme that we developed is more practical and advantageous: (1) It can be easily operated and quickly initialized; (2) its self-traction and control of online functions can ensure that every rule is fully verified; and (3) the amount of manual work is small, allowing technicians to complete the verification steps during their daily work.\nWe divided the entire validation into two modules\u2014correctness verification and integrity validation\u2014based on the concept of process management. Rules are the basic unit of the entire autoverification system. If basic rule verification is not performed at the beginning of the entire process, when the human\u2013machine judgment is inconsistent, it is difficult to confirm whether the problem is caused by algorithm error, execution error, or another reason, inevitably increasing the analysis workload. In contrast, if correctness verification is completed when the rules are established, the only reason for an inconsistency between man and machine during the release of the report issuance would be \"rule omission,\" requiring the technician to add only the corresponding rules.\nDuring the entire verification process, we implemented human\u2013computer interaction, which includes the following:\n\n An \"expected sense of play\": Before the laboratory personnel view the results, they already possess a logical expectation, and in the process, they establish a comparison of the rules and effects.\n The use of visual stimulation methods (red, green, and purple backgrounds): These methods can be quickly identified and relax the laboratory personnel.\n System pull: Once the verification succeeds or fails, it is automatically counted with the click of a button, which automatically opens the automatic report function. All the functions ensure that laboratory personnel, particularly those of the new generation, can derive enjoyment from completing the verification process, thus increasing its core value.[16]\nAccording to the experience of this research, the logic of the autoverification validation process is not difficult, but if it is applied to other laboratories on a large scale, the intermediate software supplier needs to develop the original autoverification system. The validation system is based on the autoverification system developed by our laboratory, so it is more compatible in adding new functions. However, as a supplementary function, it is difficult to graft to existing systems. We suggest that peers can refer to the program logic provided in this study. On the basis of the current functions, we will further strengthen the learning ability of the validation system and convert validation records into learning cases that can serve as a guide for laboratory technicians to use the autoverification function more efficiently.\n\nConclusions \nIn the two years that our online validation has been in use, there have never been any defects or reporting risks due to autoverification. We believe that for both intermediate and self-built autoverification systems, online validation is a useful tool for controlling the risks of autoverification and improving the quality of reports. The detailed process for this method can serve as reference for the development and implementation of LIS-based autoverification systems.\n\nAbbreviations \nAUTO 10-A: Autoverification of Clinical Laboratory Test Result 10-A; Approved Guideline\nCBC: Complete blood cell count\nCLSI: Institute of Clinical and Laboratory Standardization\nCRP: C-reactive protein\nLIS: Laboratory information system\nHBcAb: Hepatitis B virus core antibody\nHBeAb: Hepatitis B virus e antibody\nHBeAg: Hepatitis B virus e antigen\nHBsAb: Hepatitis B virus surface antibody\nHBsAg: Hepatitis B virus surface antigen\nHBV: Hepatitis B virus\nHPV: Human papilloma virus\nTAT: Turnaround time\n\nAcknowledgements \nWe would like to gratefully acknowledge the technicians working in the laboratory for their helpful collaboration. In addition, we thank Xinyu Li and Jiazhen Ren for their technical support.\n\nContributions \nAll of the authors had full access to all of the data in the study and taking responsibility for the content of the manuscript. RT conceived and designed the study. DJ, QW, BJL, DZP and JJW performed the case and sample collection, analysis, and interpretation of the data. YTC, XYD and NXM performed the analysis with constructive discussions.DJ wrote the first draft of the paper. RT reviewed and approved the final manuscript. All authors have read and approved the final manuscript.\n\nFunding \nThis research was supported by Guangdong Medical Science and Technology Research Fund (Program Grant A2020597). The funding bodies were not involved in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.\n\nEthics approval and consent to participate \nThis study was approved by the ethics review board of KingMed Diagnostics. The study adhered to relevant guidelines and regulations. The patient consent was waived by the approving ethics review board, as utilization of anonymized history data does not require patient consent.\n\nAvailability of data and materials \nAll data generated or analyzed during this study are included in this published article. The data underlying this study are available and researchers may submit data requests to the corresponding author on reasonable request.\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nReferences \n\n\n\u2191 1.0 1.1 Li, J.; Cheng, B; Ouyang, H. et al. (2018). \"Designing and evaluating autoverification rules for thyroid function profiles and sex hormone tests\". Annals of Clinical Biochemistry 55 (2): 254\u201363. doi:10.1177\/0004563217712291. PMID 28490181.   \n\n\u2191 2.0 2.1 2.2 Wang, Z.; Peng, C.; Kang, H. et al. (2019). \"Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory\". BMC Medical Informatics and Decision Making 19 (1): 123. doi:10.1186\/s12911-019-0848-2. PMC PMC6609390. PMID 31269951. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6609390 .   \n\n\u2191 Wu, J.; Pan, M.; Ouyang, H. et al. (2018). \"Establishing and Evaluating Autoverification Rules with Intelligent Guidelines for Arterial Blood Gas Analysis in a Clinical Laboratory\". SLS Technology 23 (6): 631\u201340. doi:10.1177\/2472630318775311. PMID 29787327.   \n\n\u2191 4.0 4.1 4.2 Randell, E.W.; Short, G; Lee, N. et al. (2018). \"Strategy for 90% autoverification of clinical chemistry and immunoassay test results using six sigma process improvement\". Data in Brief 18: 1740-1749. doi:10.1016\/j.dib.2018.04.080. PMC PMC5998219. PMID 29904674. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5998219 .   \n\n\u2191 Randell, E.W.; Short, G; Lee, N. et al. (2018). \"Autoverification process improvement by Six Sigma approach: Clinical chemistry & immunoassay\". Clinical Biochemistry 55: 42\u20138. doi:10.1016\/j.clinbiochem.2018.03.002. PMID 29518383.   \n\n\u2191 College of American Pathologists (21 August 2017). \"Laboratory General Checklist - CAP Accreditation Program\" (PDF). https:\/\/elss.cap.org\/elss\/ShowProperty?nodePath=\/UCMCON\/Contribution%20Folders\/DctmContent\/education\/OnlineCourseContent\/2017\/LAP-TLTM\/checklists\/cl-gen.pdf .   \n\n\u2191 \"ISO 15189:2012 Medical laboratories \u2014 Requirements for quality and competence\". International Organization for Standardization. November 2012. https:\/\/www.iso.org\/standard\/56115.html .   \n\n\u2191 Palmieri, R.; Falbo, R.; Caoowllini, F. et al. (2018). \"The development of autoverification rules applied to urinalysis performed on the AutionMAX-SediMAX platform\". Clinica Chimica Acta 485: 275\u201381. doi:10.1016\/j.cca.2018.07.001. PMID 29981288.   \n\n\u2191 Sediq, A.M.-E., Abdel-Azeez, A.G.H. (2014). \"Designing an autoverification system in Zagazig University Hospitals Laboratories: Preliminary evaluation on thyroid function profile\". Annals of Saudi Medicine 34 (5): 427\u201332. doi:10.5144\/0256-4947.2014.427. PMC PMC6074554. PMID 25827700. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6074554 .   \n\n\u2191 \"WS\/T 616-2018 (WST 616-2018)\". Chinese Standard. 20 August 2018. https:\/\/www.chinesestandard.net\/PDF\/English.aspx\/WST616-2018 .   \n\n\u2191 \"AUTO10 Autoverification of Clinical Laboratory Test Results, 1st Edition\". Clinical and Laboratory Standards Institute. 31 October 2006. https:\/\/clsi.org\/standards\/products\/automation-and-informatics\/documents\/auto10\/ .   \n\n\u2191 van Rossum, H.H. (2020). \"An approach to selecting auto-verification limits and validating their error detection performance independently for pre-analytical and analytical errors\". Clinica Chimica Acta 508: 130\u20136. doi:10.1016\/j.cca.2020.05.026. PMID 32416173.   \n\n\u2191 Krasowski, M.D.; Davis, S.R.; Drees, D. et al. (2014). \"Autoverification in a core clinical chemistry laboratory at an academic medical center\". Journal of Pathology Informatics 5 (1): 13. doi:10.4103\/2153-3539.129450. PMC PMC4023033. PMID 24843824. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4023033 .   \n\n\u2191 Jones, J.B. (2013). \"A strategic informatics approach to autoverification\". Clinics in Laboratory Medicine 33 (1): 161\u201381. doi:10.1016\/j.cll.2012.11.004. PMID 23331736.   \n\n\u2191 Fu, Q.; Ye, C.; Han, B. et al. (2020). \"Designing and Validating Autoverification Rules for Hematology Analysis in Sysmex XN-9000 Hematology System\". Clinical Laboratory 66 (4). doi:10.7754\/Clin.Lab.2019.190726. PMID 32255287.   \n\n\u2191 Guidi, G.C.; Poli, G.; Bassi, A. et al. (2009). \"Development and implementation of an automatic system for verification, validation and delivery of laboratory test results\". Clinical Chemistry and Laboratory Medicine 47 (11): 1355\u201360. doi:10.1515\/CCLM.2009.316. PMID 19817645.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. For this version, a definition of \"autoverification\" was added to the introductory sentence of the background.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\">https:\/\/www.limswiki.org\/index.php\/Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on data qualityLIMSwiki journal articles on health informaticsLIMSwiki journal articles on laboratory informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 10 June 2021, at 18:33.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 133 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","cd03d5b8e3a613ce4ec49dfb1dc3ff59_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: For <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_informatics\" title=\"Laboratory informatics\" class=\"wiki-link\" data-key=\"00edfa43edcde538a695f6d429280301\">laboratory informatics<\/a> applications, <a href=\"https:\/\/www.limswiki.org\/index.php\/Software_verification_and_validation\" title=\"Software verification and validation\" class=\"wiki-link\" data-key=\"14bdd953b2bde31c520ba57b1002195e\">validation<\/a> of the autoverification function is one of the critical steps to confirm its effectiveness before use. It is crucial to verify whether the programmed algorithm follows the expected logic and produces the expected results. This process has always relied on the assessment of human\u2013machine consistency and is mostly a manually recorded and time-consuming activity with inherent subjectivity and arbitrariness that cannot guarantee a comprehensive, timely, and continuous effectiveness evaluation of the autoverification function. To overcome these inherent limitations, we independently developed and implemented a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS)-based validation system for autoverification.\n<\/p><p><b>Methods<\/b>: We developed a correctness verification and integrity validation method (hereinafter referred to as the \"new method\") in the form of a human\u2013machine dialog. The system records personnel review steps and determines whether the human\u2013machine review results are consistent. <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">Laboratory<\/a> personnel then analyze the reasons for any inconsistency according to system prompts, add to or modify rules, reverify, and finally improve the accuracy of autoverification.\n<\/p><p><b>Results<\/b>: The validation system was successfully established and implemented. For a dataset consisting of 833 rules for 30 assays, 782 rules (93.87%) were successfully verified in the correctness verification phase, and 51 rules were deleted due to execution errors. In the integrity validation phase, 24 projects were easily verified, while the other six projects still required the additional rules or changes to the rule settings. Taking the Hepatitis B virus test as an example, from the setting of 65 rules to the automated releasing of 3,000 reports, the validation time was reduced from 452 (manual verification) to 275 hours (new method), a reduction in validation time of 177 hours. Furthermore, 94.6% (168\/182) of laboratory users believed the new method greatly reduced the workload, effectively controlled the report risk, and felt satisfied. Since 2019, over 3.5 million reports have been automatically reviewed and issued without a single clinical complaint.\n<\/p><p><b>Conclusion<\/b>: To the best of our knowledge, this is the first report to realize autoverification validation as a human\u2013machine interaction. The new method effectively controls the risks of autoverification, shortens time consumption, and improves the efficiency of laboratory verification.\n<\/p><p><b>Keywords<\/b>: autoverification, correctness verification, integrity validation, human\u2013computer interaction, risk management, laboratory information system\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Autoverification\u2014the use of automated computer-based rules to initially validate <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> test results<sup id=\"rdp-ebb-cite_ref-LiDesigning18_1-0\" class=\"reference\"><a href=\"#cite_note-LiDesigning18-1\">[1]<\/a><\/sup>\u2014is a powerful tool for the batch processing of test results and has been widely used in recent years. It has obvious advantages in reducing reporting errors, shortening turnaround time (TAT), and improving audit efficiency.<sup id=\"rdp-ebb-cite_ref-LiDesigning18_1-1\" class=\"reference\"><a href=\"#cite_note-LiDesigning18-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WangDesign19_2-0\" class=\"reference\"><a href=\"#cite_note-WangDesign19-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WuEstablish18_3-0\" class=\"reference\"><a href=\"#cite_note-WuEstablish18-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RandellStrat18_4-0\" class=\"reference\"><a href=\"#cite_note-RandellStrat18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RandellAuto18_5-0\" class=\"reference\"><a href=\"#cite_note-RandellAuto18-5\">[5]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Current_status_and_challenges\">Current status and challenges<\/span><\/h3>\n<p>Our self-developed autoverification system has been used for six years in many disciplines, such as biochemistry, immunology, hematology, microbiology, <a href=\"https:\/\/www.limswiki.org\/index.php\/Molecular_diagnostics\" title=\"Molecular diagnostics\" class=\"wiki-link\" data-key=\"8fc14cae7a6fbac9a53fae1394fae7ee\">molecular diagnostics<\/a>, and pathology. To date, 25,487 rules have been set. The system judges test results 1.1 million times a day and provides audit recommendations for 250,000 report forms, accounting for 87% of the total number of report forms. Approximately 80,000 reports are automatically generated every day. To ensure the effectiveness and safety of the autoverification system, its <a href=\"https:\/\/www.limswiki.org\/index.php\/Software_verification_and_validation\" title=\"Software verification and validation\" class=\"wiki-link\" data-key=\"14bdd953b2bde31c520ba57b1002195e\">validation<\/a> process is very important. The College of American Pathologists' laboratory accreditation checklist item GEN.43875<sup id=\"rdp-ebb-cite_ref-CAPLab17_6-0\" class=\"reference\"><a href=\"#cite_note-CAPLab17-6\">[6]<\/a><\/sup> and International Organization for Standardization's <a href=\"https:\/\/www.limswiki.org\/index.php\/ISO_15189\" title=\"ISO 15189\" class=\"wiki-link\" data-key=\"e7867fe884a6e63d87c5a1bff5c28bc2\">ISO 15189:2012<\/a> requirement 5.9.2b<sup id=\"rdp-ebb-cite_ref-ISO15189:2012_7-0\" class=\"reference\"><a href=\"#cite_note-ISO15189:2012-7\">[7]<\/a><\/sup> both require that autoverification systems undergo functional verification before use.\n<\/p><p>According to published studies, in laboratories that use autoverification, the majority of laboratories have performed personnel-based and automatic system audits with the same results\u2014manually recorded consistency\u2014and reached a conclusion after a statistical analysis of the results.<sup id=\"rdp-ebb-cite_ref-WangDesign19_2-1\" class=\"reference\"><a href=\"#cite_note-WangDesign19-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RandellStrat18_4-1\" class=\"reference\"><a href=\"#cite_note-RandellStrat18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PalmieriTheDev18_8-0\" class=\"reference\"><a href=\"#cite_note-PalmieriTheDev18-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SediqDesign14_9-0\" class=\"reference\"><a href=\"#cite_note-SediqDesign14-9\">[9]<\/a><\/sup> The manual verification method is less difficult to operate but has the following limitations:\n<\/p><p><br \/>\n<\/p>\n<ol><li> Massive validation workload: Based on the requirements of WS\/T 616-2018 (a China Health Organization recommended standard)<sup id=\"rdp-ebb-cite_ref-WS.2FT616-2018_10-0\" class=\"reference\"><a href=\"#cite_note-WS.2FT616-2018-10\">[10]<\/a><\/sup> for validation of the autoverification of quantitative <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_laboratory\" title=\"Clinical laboratory\" class=\"wiki-link\" data-key=\"307bcdf1bdbcd1bb167cee435b7a5463\">clinical laboratory<\/a> test results, every test and every sample type involved in the autoverification procedure should be tested; the validation time should be no less than three months and\/or the number of reports released should be no less than 50,000; and periodic verification should be performed every year for no less than 10 working days and\/or for no less than 5,000 reports. The validation workload is large, and it is difficult to rely on manual comparison and recording, which greatly increases the post-analytical workload.<\/li>\n<li> Reporting risk: During manual verification, personnel are prone to inertia or judgment errors. The lack of a system control mechanism for this kind of validation can generate reporting risks and directly affect clinical diagnosis and treatment.<sup id=\"rdp-ebb-cite_ref-WangDesign19_2-2\" class=\"reference\"><a href=\"#cite_note-WangDesign19-2\">[2]<\/a><\/sup><\/li><\/ol>\n<p>Therefore, there is an urgent need to design a verification method that minimizes the workload and systematically controls risks. We report a rule verification system with a small workload and ease of operation that can be used as a reference for self-built and automatic test auditing for laboratories and manufacturers.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"System_design\">System design<\/span><\/h3>\n<p>Based on the Clinical and Laboratory Standards Institute's (CLSI) AUTO10 standard<sup id=\"rdp-ebb-cite_ref-CLSIAUTO10_11-0\" class=\"reference\"><a href=\"#cite_note-CLSIAUTO10-11\">[11]<\/a><\/sup> and current review processes, we established an autoverification system including 11 rule categories. Technicians set the rules according to audit requirements and rule categories. Each item can set multiple rules, including limited range check, combined mode judgment, Delta check, sampling time validity judgment, <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">sample<\/a> abnormality judgment (e.g., hemolysis, lipemia), and <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> check. The autoverification system determines whether the report is abnormal according to the rules. Tests that do not trigger contradiction mode are displayed in green, while failed tests (triggering rules, contradictory modes set by the rules) are displayed in red, and the cause of the contradiction is indicated. If all the tests in the report are green, the <a href=\"https:\/\/www.limswiki.org\/index.php\/Barcode\" title=\"Barcode\" class=\"wiki-link\" data-key=\"e0952b5b262392be0995237aec36d355\">barcode<\/a> of the report is also green. If any test in the report is red, the report shows a red barcode, which signals a warning in the system.\n<\/p><p>According to the above steps, the autoverification system displays colors and abnormal prompts after judging the rules in a process called automatic early warning. The automatic warning is only for judgment and is not involved in the decision to issue a report. Based on this, the system automatically sends out a report with a green barcode in a process called automated reporting. Automatic early warning and automatic reporting comprise autoverification. This system is especially useful in the review of complex diagnostic projects (e.g., molecular diagnostics, pathological testing). These projects prompt absurd values from personnel. For some moderately complex projects (e.g., biochemical, blood), the combination of report reviewing, automatic warning, and automated reporting is equivalent to the autoverification system in a large number of literature reports and <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">laboratory information system<\/a> (LIS) automatic reports. The autoverification process used by our laboratory is shown in Fig. 1.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Jin_BMCMedInfoDecMak21_21.png\" class=\"image wiki-link\" data-key=\"e6ec33ca682d935fb3dec3d99af339bb\"><img alt=\"Fig1 Jin BMCMedInfoDecMak21 21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8f\/Fig1_Jin_BMCMedInfoDecMak21_21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 1.<\/b> The autoverification process. Single test results must meet all the warning rules at the same time. The autoverification algorithm can identify those samples requiring manual review that do not meet the laboratory\u2019s criteria for autoverification. If the automated reporting switch is not activated, then reports that pass the automatic warning step are manually issued. If the automated reporting switch is turned on and all tests on the report pass their warning rules, then the system automatically releases the report.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Validation_scheme\">Validation scheme<\/span><\/h3>\n<p>On the premise that automatic audits are divided into automatic warnings and automatic reports, we divide the verification system into two stages. The first stage is called correctness verification, which verifies that the operation of the rules is consistent with the expectations set by the personnel. If there is a problem, the responsible party may be the program development department. The second stage is called integrity validation. Based on the results from the first stage, this stage verifies whether the set rules include all the elements from the personnel\u2019s audit report. The functional design of the two-stage system is shown in Table 1.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 1.<\/b> Two validation methods designed for two parts of the autoverification system\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Phase\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Object\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Validation method\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Explanation\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Inconsistent solutions\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Automatic warning\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Warning rules\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Correctness verification\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">To verify that the warning rules behave as expected and produce the expected outcome\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">If the warning rule setting is wrong, delete and reset the rules\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Automated reporting\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Laboratory tests\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Integrity validation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">To confirm that the laboratory test results that pass the automatic warning can be reported automatically\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Add more warning rules according to the laboratory report criteria\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Correctness_verification\">Correctness verification<\/span><\/h3>\n<p>The correctness verification phase confirms whether the execution of a single rule is correct. It is implemented as follows: (1) For newly added rules, the system adds the label \"Pending Verification.\" (2) When the report is reviewed, the system displays the rule judgment result, and a purple color block is displayed to remind the staff to judge whether the execution result of the \"Pending Verification\" rule is correct. (3) The staff input the judgment result. (4) The system changes the rule status according to the staff input. If it is consistent, the rule label is set to \"verified,\" prompting the personnel to continue to the next stage of verification. If it is inconsistent, the staff is prompted to delete the rule. Figure 2 is a diagram representing this correctness verification process using the example of C-reactive protein (CRP). Figure 3 shows an example of the correctness verification interface.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Jin_BMCMedInfoDecMak21_21.png\" class=\"image wiki-link\" data-key=\"00c5287f7c366f618b0eecf69813d446\"><img alt=\"Fig2 Jin BMCMedInfoDecMak21 21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/54\/Fig2_Jin_BMCMedInfoDecMak21_21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 2.<\/b> Schematic diagram of the correctness verification using the example of C-reactive protein (CRP). The CRP test result was 1.8 mg\/l and passed quality control. The autoverification system searched all the rules for the CRP and hit two of them, No. 001879 and No. 002009. The No. 001879 rule (verified) checks whether the CRP result has passed the quality control. The No. 002009 rule (pending verification) intercepts the results greater than or equal to 5. Therefore, when No. 002009 is triggered, the warning information of the sample appears purple, indicating that the technician needs to confirm whether the warning result is consistent with the manual judgment. In the correctness verification interface (as shown in the subsequent Fig. 3), the system provides two options, the human\u2013machine judgment is consistent or the system judges incorrectly. The technician can confirm that the rule is performing correctly and change its status to \u201cverified.\u201d<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Jin_BMCMedInfoDecMak21_21.png\" class=\"image wiki-link\" data-key=\"e958a204e51a4c473f01d91cde6ea399\"><img alt=\"Fig3 Jin BMCMedInfoDecMak21 21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/37\/Fig3_Jin_BMCMedInfoDecMak21_21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 3.<\/b> Correctness verification interface. The result of CRP passes automatic warning according to the No.002009 rule and displays green. The technician judges whether the automated warning operates correctly.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Integrety_validation\">Integrety validation<\/span><\/h3>\n<p>Integrity validation can be started only after the correctness verification of all rules of a project is completed. It is implemented as follows. (1) After the report shows the result of the automatic warning, if the system detects that the report has been changed, a dialog box will pop up and ask the reviewer to select the reason for the modification. These reasons include (a) a rule execution error, (b) a rule setting value that is inappropriate, (c) the required addition of new rules, (d) the lack of involvement of other issues related to automatic review, and (e) automatic warning and prompt modification. The LIS records the modified content and the reasons for personnel analysis. (2) If the laboratory wants to implement automated reporting, a validation number, such as 5,000, can be set according to the complexity of the project review. (3) If the automatic warning result of the report is green (approved), the personnel will issue the report directly, and the validation number of the report will automatically increase by one. (4) If the validation number of all items on the report exceeds the set number, the report will be automatically released. (5) If the automatic warning result of the report is green (approved), but the result is modified, with the reason for the modification specified as any of a, b, or c, then the LIS will clear the validation number for the related items and stop automated reporting. Figure 4 shows the integrity validation process. The validation goals and validation amount for six projects are shown in Fig. 5.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Jin_BMCMedInfoDecMak21_21.png\" class=\"image wiki-link\" data-key=\"35ab5033c51e44847ec87c03a35feb34\"><img alt=\"Fig4 Jin BMCMedInfoDecMak21 21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/94\/Fig4_Jin_BMCMedInfoDecMak21_21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 4.<\/b> The integrity validation process.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Jin_BMCMedInfoDecMak21_21.png\" class=\"image wiki-link\" data-key=\"ac7dbbff01cae155175260df5eb5ef33\"><img alt=\"Fig5 Jin BMCMedInfoDecMak21 21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/95\/Fig5_Jin_BMCMedInfoDecMak21_21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 5.<\/b> Integrity validation target number settings and recording interface. The validation targets of the six projects in the above figure are all 3000, and the validation number is between 1900 and 2500. The corresponding reports cannot be released automatically.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Accuracy_guarantee\">Accuracy guarantee<\/span><\/h3>\n<p>The accuracy of the autoverification includes whether the rules can be identified and whether there are omissions (completeness) in the report review. Therefore, our method confirms the accuracy of the new method from these two aspects. We perform function correction and system improvement through correctness verification and integrity validation. In the validation system, we design the following logic to ensure the accuracy of the function:\n<\/p>\n<ol><li> The new rule is automatically deleted if it fails the correctness verification within 10 days.<\/li>\n<li> The rule is not allowed to be modified.<\/li>\n<li> If the rule fails the correctness verification, it is forbidden to be converted.<\/li>\n<li> If the autoverification of a single project fails the integrity validation, the historical validation amount is cleared.<\/li><\/ol>\n<h3><span class=\"mw-headline\" id=\"Data_collection\">Data collection<\/span><\/h3>\n<p>The validation data of 30 assays from October 2019 to January 2020 were collected for analysis, and in total, 833 early warning rules were obtained. A total of 926,195 reports was used to evaluate the accuracy of the new method.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Time_consumption_statistics\">Time consumption statistics<\/span><\/h3>\n<p>We used HBV as an example to introduce the comparison of the validation time before and after the new method was used. In the measurement of the validation time, we divided the complete autoverification into 10 stages. Time statistics were collected for manual verification and new method verification for each step. We used systematic records and estimates to develop time statistics for different stages.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Satisfaction_survey\">Satisfaction survey<\/span><\/h3>\n<p>We used questionnaires to evaluate the effectiveness of new methods used by laboratory technicians. The survey was launched using the online tool <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wjx.cn\/\" target=\"_blank\">WJX<\/a>, which feeds back the percentage of responses and the total numbers.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Correctness_verification_results\">Correctness verification results<\/span><\/h3>\n<p>Among the 833 rules, 782 (93.88%) were successfully verified for correctness, with a total of 3,814 validations, including 2,230 (58.47%) released tests and 1,584 (41.53%) intercepted tests. The inconsistencies were verified, and 51 (0.06%) error rules were deleted. The reasons for verification failure are shown in Table 2.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 2.<\/b> List of reasons for correctness verification failure\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Error type\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Proportion (%)\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Sample\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Solution\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Human error\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">63.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Incorrect English letter case in the text of the rules, resulting in no warning\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reset the rules\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Specific warning target\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Early warning of diagnostic results and microscopy results in a special report interface for pathology\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Add a supplementary algorithm code\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Algorithm code error\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HPV typing results could not be verified with the Delta Check; the results of the microbial project identification could not be correlated with a variety of drug sensitivity combinations\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fix the algorithm code\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Software compatibility problem\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Problem with the precision of the number comparison script\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fix the algorithm code\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Integrity_validation_results\">Integrity validation results<\/span><\/h3>\n<p>We collected integrity validation data through system export and department feedback. The reasons provided for rule modification were automatic warning prompt and rule modification (5, 10.6%), rule execution error (0, 0%), improper setting values (15, 31.9%), new rule added (18, 38.3%), and no automatic warning involving other questions (9, 19.2%). The integrity of all projects was verified within one month, and the problems found are shown in Table 3.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 3.<\/b> List of reasons why integrity validation failed\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Test\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Reason for not passing\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Solution\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HPV genotyping\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">There was no comprehensive analysis of the combined thin-layer cytology results\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Analyze the results associated with thin-layer cytology\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Urea\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The limit range was too wide\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reduce the limit range\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Albumin\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Review of the detection system produces an error\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Specify the detection system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Test results were checked only on the same day as the barcode\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Extend the backdating of the historical results\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HBsAg HBsAb HBeAg HBeAb HBcAb\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not all composite mode scenarios were covered\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Add a joint audit of the portfolio project results\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cortisol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">There was no warning of abnormal rhythms\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Add a rule about checking sampling time\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Comparison_of_the_two_methods\">Comparison of the two methods<\/span><\/h3>\n<p>The comparison of manual record analysis and the new method for different steps is shown in Table 4. The new method performs four automation steps, reduces the personnel workload, and automatically controls the enabling and disabling of automatic report release through system monitoring report modification. The increased accuracy verification can quickly eliminate rule setting exceptions and development loopholes while reducing the time needed for personnel analysis. The manual record analysis and the new method took 452 hours and 275 hours to complete, respectively.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 4.<\/b> Comparison of the time consumption (hours) of the two methods for verifying HBV reports for 3,000 cases. In the measurement of the validation time, we divided the complete autoverification into 10 stages. In steps 4\u20136, in total, 3,000 reports are used for statistics. The time consumption of the consistent work content in the new and old methods is subject to the following: the manual timing of the old method, such as steps 1, 2, 4, and 9; the inconsistent steps in the two methods; the new steps that are recorded in the system, such as step 3; the saving step time clearing, such as steps 5, 6, 7, and 10; and the remaining steps that are estimated, such as step 8. For automatic implementation, the time is calculated as zero. <sup>a<\/sup>Reasons for invalid locking rules. <sup>b<\/sup>Reduced workload. <sup>c<\/sup>Controlled risks.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Steps\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Manual validation (h)\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">New method (h)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1. Set 65 rules\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2. Perform Rule 130 test\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3. Correctness verification\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.25<sup>a<\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4. Personnel comparison report and results review\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">240\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">240\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5. Record comparison result\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0<sup>b<\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6. Analysis of the verification number\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0<sup>b<\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7. Determine whether to activate automatic approval\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0<sup>b<\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8. Personnel analysis of the reasons for inconsistent audit results\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">90\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9. Add and modify rules\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10. Determine whether to turn off autoverification\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0<sup>c<\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Total<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">452\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">275\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Satisfaction_survey_2\">Satisfaction survey<\/span><\/h3>\n<p>After using the new method for one year, we conducted a satisfaction survey of laboratory personnel who used the function. We distributed 182 questionnaires and recovered 168 of them, with a response rate of 92.3%. The survey results showed that 94.6% of laboratory users believed that the new method could greatly reduce the workload, effectively control the report risk, and produce satisfactory or very satisfactory assessments of the new method.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>The core of the use of autoverification lies in the validation of system functions and rules. Due to the complexity of these rules, it is impossible to find all the functional defects by relying solely on function validation before the system goes online, and even human input errors cannot be carried out in the validation.<sup id=\"rdp-ebb-cite_ref-VanRossumAnApp20_12-0\" class=\"reference\"><a href=\"#cite_note-VanRossumAnApp20-12\">[12]<\/a><\/sup> Such functional defects must be found in actual application scenarios with multiple different rule settings, such as the incorrect input of full-width symbols, that is, correctness verification. Furthermore, the premise of rule verification is to include a review of the logic of all reviewers in the system, which can be discovered only in actual application scenarios. Additionally, integrity validation can be performed in actual application scenarios to truly find problems.<sup id=\"rdp-ebb-cite_ref-KrasowskiAuto14_13-0\" class=\"reference\"><a href=\"#cite_note-KrasowskiAuto14-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JonesAStrat13_14-0\" class=\"reference\"><a href=\"#cite_note-JonesAStrat13-14\">[14]<\/a><\/sup>\n<\/p><p>We initially designed the system in two parts\u2014automatic warning and automated reporting\u2014to allow complex detection items (molecular and pathological examinations, final human reports, and system prompt errors) to be included in the automatic review. Laboratory technicians could then choose to address the needs of different measurements. These two parts correspond to two verification steps: the automatic warning portion that performs correctness verification, and the automated reporting portion that performs integrity validation.\n<\/p><p>Compared with other systems reported in the literature<sup id=\"rdp-ebb-cite_ref-RandellStrat18_4-2\" class=\"reference\"><a href=\"#cite_note-RandellStrat18-4\">[4]<\/a><\/sup>, the advantages of the new method are mainly simplifying the verification process, reducing the verification workload, and ensuring the accuracy of the verification results. As shown in Fig. 6, in the manual verification scheme, the junior staff will review the results, and then the intermediate staff will repeat reviewing the results and combine the autoverification results to determine the human\u2013machine consistency to complete the autoverification function validation. The whole process focuses on human leadership. The new method uses system monitoring to judge the accuracy of the autoverification based on the operational trajectories of different personnel. By interacting with personnel, the system collects validation data and controls the operation of autoverification.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Jin_BMCMedInfoDecMak21_21.png\" class=\"image wiki-link\" data-key=\"66a18e4c219a398bff565d602f4a96f9\"><img alt=\"Fig6 Jin BMCMedInfoDecMak21 21.png\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/ab\/Fig6_Jin_BMCMedInfoDecMak21_21.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 6.<\/b> Schematic diagram of the process comparison between the manual method and the new method<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>We delineate the advantages of the new method, compared to manual validation, in Table 5.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 5.<\/b> Comparison of the advantages of the new method and manual verification\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Advantages\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Difference\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Manual validation\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">New method\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Explanation\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Efficiency improvement\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Whether to add extra workload?\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">YES\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NO\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No additional personnel are required to manually record the reason for the inconsistency. The new method is that the system completes judgment and records while personnel review the reports normally. The system will control the operation of the autoverification program based on the consistency results.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Can the cause of inconsistency be quickly determined?\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NO\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">YES\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The main reasons for the inconsistency are abnormal rule settings and lack of necessary rules. The new method correspondingly sets up correctness verification and integrity validation for these two main reasons. In different verification stages, only the main reason for that stage can be traced back.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">Risk control\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Is it possible to skip the validation process?\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">YES\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NO\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Starting from setting the rules, the system will pull the validation process, and no validation link can be skipped.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Whether to ensure sufficient amount of validation data?\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NO\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">YES\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">In the process of normal personnel issuance, the system will truthfully record the validation data. Before the set data volume is reached, the automated reporting function is prohibited.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Can autoverification be used in the case of failed validation?\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">YES\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NO\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">When the system confirms that the validation fails due to a defect in the autoverification, it will prohibit the rule conversion or the automated reporting from being enabled.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Compared with the traditional method, the true positives and false positives of the \"personal and machine-based audit results\" are easy to understand, but if the indicators are abnormal, it can be difficult to find the cause of this abnormality, especially after all the reasons are verified after thousands of reports are released.<sup id=\"rdp-ebb-cite_ref-FuDesign20_15-0\" class=\"reference\"><a href=\"#cite_note-FuDesign20-15\">[15]<\/a><\/sup> Consequently, the audit scenario has become blurred in the auditor's memory, and it becomes inefficient to check the problems one by one. The process-based validation scheme that we developed is more practical and advantageous: (1) It can be easily operated and quickly initialized; (2) its self-traction and control of online functions can ensure that every rule is fully verified; and (3) the amount of manual work is small, allowing technicians to complete the verification steps during their daily work.\n<\/p><p>We divided the entire validation into two modules\u2014correctness verification and integrity validation\u2014based on the concept of process management. Rules are the basic unit of the entire autoverification system. If basic rule verification is not performed at the beginning of the entire process, when the human\u2013machine judgment is inconsistent, it is difficult to confirm whether the problem is caused by algorithm error, execution error, or another reason, inevitably increasing the analysis workload. In contrast, if correctness verification is completed when the rules are established, the only reason for an inconsistency between man and machine during the release of the report issuance would be \"rule omission,\" requiring the technician to add only the corresponding rules.\n<\/p><p>During the entire verification process, we implemented human\u2013computer interaction, which includes the following:\n<\/p>\n<ol><li> An \"expected sense of play\": Before the laboratory personnel view the results, they already possess a logical expectation, and in the process, they establish a comparison of the rules and effects.<\/li>\n<li> The use of visual stimulation methods (red, green, and purple backgrounds): These methods can be quickly identified and relax the laboratory personnel.<\/li>\n<li> System pull: Once the verification succeeds or fails, it is automatically counted with the click of a button, which automatically opens the automatic report function. All the functions ensure that laboratory personnel, particularly those of the new generation, can derive enjoyment from completing the verification process, thus increasing its core value.<sup id=\"rdp-ebb-cite_ref-GuidiDev09_16-0\" class=\"reference\"><a href=\"#cite_note-GuidiDev09-16\">[16]<\/a><\/sup><\/li><\/ol>\n<p>According to the experience of this research, the logic of the autoverification validation process is not difficult, but if it is applied to other laboratories on a large scale, the intermediate software supplier needs to develop the original autoverification system. The validation system is based on the autoverification system developed by our laboratory, so it is more compatible in adding new functions. However, as a supplementary function, it is difficult to graft to existing systems. We suggest that peers can refer to the program logic provided in this study. On the basis of the current functions, we will further strengthen the learning ability of the validation system and convert validation records into learning cases that can serve as a guide for laboratory technicians to use the autoverification function more efficiently.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>In the two years that our online validation has been in use, there have never been any defects or reporting risks due to autoverification. We believe that for both intermediate and self-built autoverification systems, online validation is a useful tool for controlling the risks of autoverification and improving the quality of reports. The detailed process for this method can serve as reference for the development and implementation of LIS-based autoverification systems.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>AUTO 10-A<\/b>: Autoverification of Clinical Laboratory Test Result 10-A; Approved Guideline\n<\/p><p><b>CBC<\/b>: Complete blood cell count\n<\/p><p><b>CLSI<\/b>: Institute of Clinical and Laboratory Standardization\n<\/p><p><b>CRP<\/b>: C-reactive protein\n<\/p><p><b>LIS<\/b>: Laboratory information system\n<\/p><p><b>HBcAb<\/b>: Hepatitis B virus core antibody\n<\/p><p><b>HBeAb<\/b>: Hepatitis B virus e antibody\n<\/p><p><b>HBeAg<\/b>: Hepatitis B virus e antigen\n<\/p><p><b>HBsAb<\/b>: Hepatitis B virus surface antibody\n<\/p><p><b>HBsAg<\/b>: Hepatitis B virus surface antigen\n<\/p><p><b>HBV<\/b>: Hepatitis B virus\n<\/p><p><b>HPV<\/b>: Human papilloma virus\n<\/p><p><b>TAT<\/b>: Turnaround time\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We would like to gratefully acknowledge the technicians working in the laboratory for their helpful collaboration. In addition, we thank Xinyu Li and Jiazhen Ren for their technical support.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Contributions\">Contributions<\/span><\/h3>\n<p>All of the authors had full access to all of the data in the study and taking responsibility for the content of the manuscript. RT conceived and designed the study. DJ, QW, BJL, DZP and JJW performed the case and sample collection, analysis, and interpretation of the data. YTC, XYD and NXM performed the analysis with constructive discussions.DJ wrote the first draft of the paper. RT reviewed and approved the final manuscript. All authors have read and approved the final manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research was supported by Guangdong Medical Science and Technology Research Fund (Program Grant A2020597). The funding bodies were not involved in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Ethics_approval_and_consent_to_participate\">Ethics approval and consent to participate<\/span><\/h3>\n<p>This study was approved by the ethics review board of KingMed Diagnostics. The study adhered to relevant guidelines and regulations. The patient consent was waived by the approving ethics review board, as utilization of anonymized history data does not require patient consent.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Availability_of_data_and_materials\">Availability of data and materials<\/span><\/h3>\n<p>All data generated or analyzed during this study are included in this published article. The data underlying this study are available and researchers may submit data requests to the corresponding author on reasonable request.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-LiDesigning18-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiDesigning18_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiDesigning18_1-1\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, J.; Cheng, B; Ouyang, H. et al. (2018). \"Designing and evaluating autoverification rules for thyroid function profiles and sex hormone tests\". <i>Annals of Clinical Biochemistry<\/i> <b>55<\/b> (2): 254\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F0004563217712291\" target=\"_blank\">10.1177\/0004563217712291<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28490181\" target=\"_blank\">28490181<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Designing+and+evaluating+autoverification+rules+for+thyroid+function+profiles+and+sex+hormone+tests&rft.jtitle=Annals+of+Clinical+Biochemistry&rft.aulast=Li%2C+J.%3B+Cheng%2C+B%3B+Ouyang%2C+H.+et+al.&rft.au=Li%2C+J.%3B+Cheng%2C+B%3B+Ouyang%2C+H.+et+al.&rft.date=2018&rft.volume=55&rft.issue=2&rft.pages=254%E2%80%9363&rft_id=info:doi\/10.1177%2F0004563217712291&rft_id=info:pmid\/28490181&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangDesign19-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WangDesign19_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-WangDesign19_2-1\">2.1<\/a><\/sup> <sup><a href=\"#cite_ref-WangDesign19_2-2\">2.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, Z.; Peng, C.; Kang, H. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6609390\" target=\"_blank\">\"Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>19<\/b> (1): 123. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12911-019-0848-2\" target=\"_blank\">10.1186\/s12911-019-0848-2<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6609390\/\" target=\"_blank\">PMC6609390<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31269951\" target=\"_blank\">31269951<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6609390\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6609390<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Design+and+evaluation+of+a+LIS-based+autoverification+system+for+coagulation+assays+in+a+core+clinical+laboratory&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Wang%2C+Z.%3B+Peng%2C+C.%3B+Kang%2C+H.+et+al.&rft.au=Wang%2C+Z.%3B+Peng%2C+C.%3B+Kang%2C+H.+et+al.&rft.date=2019&rft.volume=19&rft.issue=1&rft.pages=123&rft_id=info:doi\/10.1186%2Fs12911-019-0848-2&rft_id=info:pmc\/PMC6609390&rft_id=info:pmid\/31269951&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6609390&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WuEstablish18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WuEstablish18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wu, J.; Pan, M.; Ouyang, H. et al. (2018). \"Establishing and Evaluating Autoverification Rules with Intelligent Guidelines for Arterial Blood Gas Analysis in a Clinical Laboratory\". <i>SLS Technology<\/i> <b>23<\/b> (6): 631\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F2472630318775311\" target=\"_blank\">10.1177\/2472630318775311<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29787327\" target=\"_blank\">29787327<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Establishing+and+Evaluating+Autoverification+Rules+with+Intelligent+Guidelines+for+Arterial+Blood+Gas+Analysis+in+a+Clinical+Laboratory&rft.jtitle=SLS+Technology&rft.aulast=Wu%2C+J.%3B+Pan%2C+M.%3B+Ouyang%2C+H.+et+al.&rft.au=Wu%2C+J.%3B+Pan%2C+M.%3B+Ouyang%2C+H.+et+al.&rft.date=2018&rft.volume=23&rft.issue=6&rft.pages=631%E2%80%9340&rft_id=info:doi\/10.1177%2F2472630318775311&rft_id=info:pmid\/29787327&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RandellStrat18-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RandellStrat18_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-RandellStrat18_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-RandellStrat18_4-2\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Randell, E.W.; Short, G; Lee, N. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5998219\" target=\"_blank\">\"Strategy for 90% autoverification of clinical chemistry and immunoassay test results using six sigma process improvement\"<\/a>. <i>Data in Brief<\/i> <b>18<\/b>: 1740-1749. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.dib.2018.04.080\" target=\"_blank\">10.1016\/j.dib.2018.04.080<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5998219\/\" target=\"_blank\">PMC5998219<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29904674\" target=\"_blank\">29904674<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5998219\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5998219<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Strategy+for+90%25+autoverification+of+clinical+chemistry+and+immunoassay+test+results+using+six+sigma+process+improvement&rft.jtitle=Data+in+Brief&rft.aulast=Randell%2C+E.W.%3B+Short%2C+G%3B+Lee%2C+N.+et+al.&rft.au=Randell%2C+E.W.%3B+Short%2C+G%3B+Lee%2C+N.+et+al.&rft.date=2018&rft.volume=18&rft.pages=1740-1749&rft_id=info:doi\/10.1016%2Fj.dib.2018.04.080&rft_id=info:pmc\/PMC5998219&rft_id=info:pmid\/29904674&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5998219&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RandellAuto18-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RandellAuto18_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Randell, E.W.; Short, G; Lee, N. et al. (2018). \"Autoverification process improvement by Six Sigma approach: Clinical chemistry & immunoassay\". <i>Clinical Biochemistry<\/i> <b>55<\/b>: 42\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.clinbiochem.2018.03.002\" target=\"_blank\">10.1016\/j.clinbiochem.2018.03.002<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29518383\" target=\"_blank\">29518383<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Autoverification+process+improvement+by+Six+Sigma+approach%3A+Clinical+chemistry+%26+immunoassay&rft.jtitle=Clinical+Biochemistry&rft.aulast=Randell%2C+E.W.%3B+Short%2C+G%3B+Lee%2C+N.+et+al.&rft.au=Randell%2C+E.W.%3B+Short%2C+G%3B+Lee%2C+N.+et+al.&rft.date=2018&rft.volume=55&rft.pages=42%E2%80%938&rft_id=info:doi\/10.1016%2Fj.clinbiochem.2018.03.002&rft_id=info:pmid\/29518383&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CAPLab17-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CAPLab17_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">College of American Pathologists (21 August 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/elss.cap.org\/elss\/ShowProperty?nodePath=\/UCMCON\/Contribution%20Folders\/DctmContent\/education\/OnlineCourseContent\/2017\/LAP-TLTM\/checklists\/cl-gen.pdf\" target=\"_blank\">\"Laboratory General Checklist - CAP Accreditation Program\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/elss.cap.org\/elss\/ShowProperty?nodePath=\/UCMCON\/Contribution%20Folders\/DctmContent\/education\/OnlineCourseContent\/2017\/LAP-TLTM\/checklists\/cl-gen.pdf\" target=\"_blank\">https:\/\/elss.cap.org\/elss\/ShowProperty?nodePath=\/UCMCON\/Contribution%20Folders\/DctmContent\/education\/OnlineCourseContent\/2017\/LAP-TLTM\/checklists\/cl-gen.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Laboratory+General+Checklist+-+CAP+Accreditation+Program&rft.atitle=&rft.aulast=College+of+American+Pathologists&rft.au=College+of+American+Pathologists&rft.date=21+August+2017&rft_id=https%3A%2F%2Felss.cap.org%2Felss%2FShowProperty%3FnodePath%3D%2FUCMCON%2FContribution%2520Folders%2FDctmContent%2Feducation%2FOnlineCourseContent%2F2017%2FLAP-TLTM%2Fchecklists%2Fcl-gen.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO15189:2012-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISO15189:2012_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/56115.html\" target=\"_blank\">\"ISO 15189:2012 Medical laboratories \u2014 Requirements for quality and competence\"<\/a>. International Organization for Standardization. November 2012<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/56115.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/56115.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO+15189%3A2012+Medical+laboratories+%E2%80%94+Requirements+for+quality+and+competence&rft.atitle=&rft.date=November+2012&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F56115.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PalmieriTheDev18-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PalmieriTheDev18_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Palmieri, R.; Falbo, R.; Caoowllini, F. et al. (2018). \"The development of autoverification rules applied to urinalysis performed on the AutionMAX-SediMAX platform\". <i>Clinica Chimica Acta<\/i> <b>485<\/b>: 275\u201381. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cca.2018.07.001\" target=\"_blank\">10.1016\/j.cca.2018.07.001<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29981288\" target=\"_blank\">29981288<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+development+of+autoverification+rules+applied+to+urinalysis+performed+on+the+AutionMAX-SediMAX+platform&rft.jtitle=Clinica+Chimica+Acta&rft.aulast=Palmieri%2C+R.%3B+Falbo%2C+R.%3B+Caoowllini%2C+F.+et+al.&rft.au=Palmieri%2C+R.%3B+Falbo%2C+R.%3B+Caoowllini%2C+F.+et+al.&rft.date=2018&rft.volume=485&rft.pages=275%E2%80%9381&rft_id=info:doi\/10.1016%2Fj.cca.2018.07.001&rft_id=info:pmid\/29981288&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SediqDesign14-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SediqDesign14_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sediq, A.M.-E., Abdel-Azeez, A.G.H. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6074554\" target=\"_blank\">\"Designing an autoverification system in Zagazig University Hospitals Laboratories: Preliminary evaluation on thyroid function profile\"<\/a>. <i>Annals of Saudi Medicine<\/i> <b>34<\/b> (5): 427\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5144%2F0256-4947.2014.427\" target=\"_blank\">10.5144\/0256-4947.2014.427<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6074554\/\" target=\"_blank\">PMC6074554<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25827700\" target=\"_blank\">25827700<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6074554\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6074554<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Designing+an+autoverification+system+in+Zagazig+University+Hospitals+Laboratories%3A+Preliminary+evaluation+on+thyroid+function+profile&rft.jtitle=Annals+of+Saudi+Medicine&rft.aulast=Sediq%2C+A.M.-E.%2C+Abdel-Azeez%2C+A.G.H.&rft.au=Sediq%2C+A.M.-E.%2C+Abdel-Azeez%2C+A.G.H.&rft.date=2014&rft.volume=34&rft.issue=5&rft.pages=427%E2%80%9332&rft_id=info:doi\/10.5144%2F0256-4947.2014.427&rft_id=info:pmc\/PMC6074554&rft_id=info:pmid\/25827700&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6074554&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WS.2FT616-2018-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WS.2FT616-2018_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.chinesestandard.net\/PDF\/English.aspx\/WST616-2018\" target=\"_blank\">\"WS\/T 616-2018 (WST 616-2018)\"<\/a>. <i>Chinese Standard<\/i>. 20 August 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.chinesestandard.net\/PDF\/English.aspx\/WST616-2018\" target=\"_blank\">https:\/\/www.chinesestandard.net\/PDF\/English.aspx\/WST616-2018<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=WS%2FT+616-2018+%28WST+616-2018%29&rft.atitle=Chinese+Standard&rft.date=20+August+2018&rft_id=https%3A%2F%2Fwww.chinesestandard.net%2FPDF%2FEnglish.aspx%2FWST616-2018&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CLSIAUTO10-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CLSIAUTO10_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/clsi.org\/standards\/products\/automation-and-informatics\/documents\/auto10\/\" target=\"_blank\">\"AUTO10 Autoverification of Clinical Laboratory Test Results, 1st Edition\"<\/a>. Clinical and Laboratory Standards Institute. 31 October 2006<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/clsi.org\/standards\/products\/automation-and-informatics\/documents\/auto10\/\" target=\"_blank\">https:\/\/clsi.org\/standards\/products\/automation-and-informatics\/documents\/auto10\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=AUTO10+Autoverification+of+Clinical+Laboratory+Test+Results%2C+1st+Edition&rft.atitle=&rft.date=31+October+2006&rft.pub=Clinical+and+Laboratory+Standards+Institute&rft_id=https%3A%2F%2Fclsi.org%2Fstandards%2Fproducts%2Fautomation-and-informatics%2Fdocuments%2Fauto10%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VanRossumAnApp20-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VanRossumAnApp20_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">van Rossum, H.H. (2020). \"An approach to selecting auto-verification limits and validating their error detection performance independently for pre-analytical and analytical errors\". <i>Clinica Chimica Acta<\/i> <b>508<\/b>: 130\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cca.2020.05.026\" target=\"_blank\">10.1016\/j.cca.2020.05.026<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32416173\" target=\"_blank\">32416173<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+approach+to+selecting+auto-verification+limits+and+validating+their+error+detection+performance+independently+for+pre-analytical+and+analytical+errors&rft.jtitle=Clinica+Chimica+Acta&rft.aulast=van+Rossum%2C+H.H.&rft.au=van+Rossum%2C+H.H.&rft.date=2020&rft.volume=508&rft.pages=130%E2%80%936&rft_id=info:doi\/10.1016%2Fj.cca.2020.05.026&rft_id=info:pmid\/32416173&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrasowskiAuto14-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrasowskiAuto14_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krasowski, M.D.; Davis, S.R.; Drees, D. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4023033\" target=\"_blank\">\"Autoverification in a core clinical chemistry laboratory at an academic medical center\"<\/a>. <i>Journal of Pathology Informatics<\/i> <b>5<\/b> (1): 13. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2F2153-3539.129450\" target=\"_blank\">10.4103\/2153-3539.129450<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4023033\/\" target=\"_blank\">PMC4023033<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24843824\" target=\"_blank\">24843824<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4023033\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4023033<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Autoverification+in+a+core+clinical+chemistry+laboratory+at+an+academic+medical+center&rft.jtitle=Journal+of+Pathology+Informatics&rft.aulast=Krasowski%2C+M.D.%3B+Davis%2C+S.R.%3B+Drees%2C+D.+et+al.&rft.au=Krasowski%2C+M.D.%3B+Davis%2C+S.R.%3B+Drees%2C+D.+et+al.&rft.date=2014&rft.volume=5&rft.issue=1&rft.pages=13&rft_id=info:doi\/10.4103%2F2153-3539.129450&rft_id=info:pmc\/PMC4023033&rft_id=info:pmid\/24843824&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4023033&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JonesAStrat13-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JonesAStrat13_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jones, J.B. (2013). \"A strategic informatics approach to autoverification\". <i>Clinics in Laboratory Medicine<\/i> <b>33<\/b> (1): 161\u201381. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cll.2012.11.004\" target=\"_blank\">10.1016\/j.cll.2012.11.004<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23331736\" target=\"_blank\">23331736<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+strategic+informatics+approach+to+autoverification&rft.jtitle=Clinics+in+Laboratory+Medicine&rft.aulast=Jones%2C+J.B.&rft.au=Jones%2C+J.B.&rft.date=2013&rft.volume=33&rft.issue=1&rft.pages=161%E2%80%9381&rft_id=info:doi\/10.1016%2Fj.cll.2012.11.004&rft_id=info:pmid\/23331736&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FuDesign20-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FuDesign20_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fu, Q.; Ye, C.; Han, B. et al. (2020). \"Designing and Validating Autoverification Rules for Hematology Analysis in Sysmex XN-9000 Hematology System\". <i>Clinical Laboratory<\/i> <b>66<\/b> (4). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7754%2FClin.Lab.2019.190726\" target=\"_blank\">10.7754\/Clin.Lab.2019.190726<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32255287\" target=\"_blank\">32255287<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Designing+and+Validating+Autoverification+Rules+for+Hematology+Analysis+in+Sysmex+XN-9000+Hematology+System&rft.jtitle=Clinical+Laboratory&rft.aulast=Fu%2C+Q.%3B+Ye%2C+C.%3B+Han%2C+B.+et+al.&rft.au=Fu%2C+Q.%3B+Ye%2C+C.%3B+Han%2C+B.+et+al.&rft.date=2020&rft.volume=66&rft.issue=4&rft_id=info:doi\/10.7754%2FClin.Lab.2019.190726&rft_id=info:pmid\/32255287&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuidiDev09-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuidiDev09_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Guidi, G.C.; Poli, G.; Bassi, A. et al. (2009). \"Development and implementation of an automatic system for verification, validation and delivery of laboratory test results\". <i>Clinical Chemistry and Laboratory Medicine<\/i> <b>47<\/b> (11): 1355\u201360. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2FCCLM.2009.316\" target=\"_blank\">10.1515\/CCLM.2009.316<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19817645\" target=\"_blank\">19817645<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+and+implementation+of+an+automatic+system+for+verification%2C+validation+and+delivery+of+laboratory+test+results&rft.jtitle=Clinical+Chemistry+and+Laboratory+Medicine&rft.aulast=Guidi%2C+G.C.%3B+Poli%2C+G.%3B+Bassi%2C+A.+et+al.&rft.au=Guidi%2C+G.C.%3B+Poli%2C+G.%3B+Bassi%2C+A.+et+al.&rft.date=2009&rft.volume=47&rft.issue=11&rft.pages=1355%E2%80%9360&rft_id=info:doi\/10.1515%2FCCLM.2009.316&rft_id=info:pmid\/19817645&rfr_id=info:sid\/en.wikipedia.org:Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. For this version, a definition of \"autoverification\" was added to the introductory sentence of the background.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150557\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.488 seconds\nReal time usage: 2.604 seconds\nPreprocessor visited node count: 13774\/1000000\nPreprocessor generated node count: 31544\/1000000\nPost\u2010expand include size: 116451\/2097152 bytes\nTemplate argument size: 38678\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 296.551 1 - -total\n 80.54% 238.832 1 - Template:Reflist\n 66.72% 197.867 16 - Template:Citation\/core\n 57.12% 169.402 12 - Template:Cite_journal\n 13.23% 39.235 4 - Template:Cite_web\n 12.64% 37.496 1 - Template:Infobox_journal_article\n 12.04% 35.690 1 - Template:Infobox\n 9.33% 27.659 28 - Template:Citation\/identifier\n 8.29% 24.583 80 - Template:Infobox\/row\n 3.64% 10.782 17 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12592-0!*!0!!en!5!* and timestamp 20210705150554 and revision id 43049\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results\">https:\/\/www.limswiki.org\/index.php\/Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","cd03d5b8e3a613ce4ec49dfb1dc3ff59_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/8\/8f\/Fig1_Jin_BMCMedInfoDecMak21_21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/5\/54\/Fig2_Jin_BMCMedInfoDecMak21_21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/3\/37\/Fig3_Jin_BMCMedInfoDecMak21_21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/94\/Fig4_Jin_BMCMedInfoDecMak21_21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/95\/Fig5_Jin_BMCMedInfoDecMak21_21.png","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/a\/ab\/Fig6_Jin_BMCMedInfoDecMak21_21.png"],"cd03d5b8e3a613ce4ec49dfb1dc3ff59_timestamp":1625497554,"236fa37968f1db59b5299967fce1aa76_type":"article","236fa37968f1db59b5299967fce1aa76_title":"Diagnostic informatics: The role of digital health in diagnostic stewardship and the achievement of excellence, safety, and value (Georgiou et al. 2021)","236fa37968f1db59b5299967fce1aa76_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value","236fa37968f1db59b5299967fce1aa76_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Diagnostic informatics: The role of digital health in diagnostic stewardship and the achievement of excellence, safety, and value\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDiagnostic informatics: The role of digital health in diagnostic stewardship and the achievement of excellence, safety, and valueJournal\n \nFrontiers in Digital HealthAuthor(s)\n \nGeorgiou, Andrew; Li, Julie; Hardie, Rae-Anne; Wabe, Nasir; Horvath, Andrea R.; Post, Jeffrey J.; Eigenstetter, Alex; Lindeman, Robert; Lam, Que; Badrick, Tony; Pearce, ChristopherAuthor affiliation(s)\n \nMacquarie University, Prince of Wales Hospital, University of New South Wales, New South Wales Health Pathology, Austin Health, Royal College of Pathologists of Australasia, Outcome Health, Monash UniversityPrimary contact\n \nEmail: andrew dot georgiou at mq dot edu dot auEditors\n \nChapman, WendyYear published\n \n2021Volume and issue\n \n3Article #\n \n659652DOI\n \n10.3389\/fdgth.2021.659652ISSN\n \n2673-253XDistribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2021.659652\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2021.659652\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Diagnostic error and the establishment of outcome-based diagnostic informatics research \n4 Enhancing the safety and effectiveness of test result management and follow-up \n5 Digitally enabled and enhanced decision support \n6 Discussion \n7 Acknowledgements \n\n7.1 Author contributions \n7.2 Funding \n7.3 Conflict of interest \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nDiagnostic investigations (i.e., pathology laboratory analysis and medical imaging) aim to increase the certainty of the presence of or absence of disease by supporting the process of differential diagnosis, support clinical management, and monitor a patient's trajectory (e.g., disease progression or response to treatment). Digital health\u2014defined as the collection, storage, retrieval, transmission, and utilization of data, information, and knowledge to support healthcare\u2014has become an essential component of the diagnostic investigational process, helping to facilitate the accuracy and timeliness of information transfer and enhance the effectiveness of decision-making processes. Digital health is also important to diagnostic stewardship, which involves coordinated guidance and interventions to ensure the appropriate utilization of diagnostic tests for therapeutic decision-making. Diagnostic stewardship and informatics are thus important in efforts to establish shared decision-making. This is because they contribute to the establishment of shared information platforms (which enable patients to read, comment on, and share in decisions about their care) based on timely and meaningful communication. \nThis paper will outline key diagnostic informatics and stewardship initiatives across three interrelated fields: (1) diagnostic error and the establishment of outcome-based diagnostic research, (2) the safety and effectiveness of test result management and follow-up, and (3) digitally enhanced clinical decision support systems.\nKeywords: pathology, diagnostic error, outcomes based assessment, evaluation, safety, health informatics, decision support, test result follow-up\n\nIntroduction \nDiagnostic investigations and tests (e.g., pathology and medical imaging) involve the observation of personal characteristics, symptoms, signs, and history. Their aim is to increase certainty of the presence of or absence of disease (or in the case of the differential diagnosis process, distinguish between different possible diagnoses), support clinical management, and monitor a patient's trajectory (e.g., during or after treatment).[1] Most recently, diagnostic investigations have had a pivotal role in the response to the SARS-CoV-2 pandemic through laboratory testing and epidemiological surveillance of viral infection, not only from its detection and diagnosis (e.g., via molecular testing by real-time polymerase chain reaction [RT-PCR]) but also its prognostication, treatment, monitoring (e.g., testing related to comorbidities), and recovery (aided by anti-SARS-CoV-2 monoclonal antibodies).\nDigital health\u2014defined as the collection, storage, retrieval, transmission, and utilization of data, information, and knowledge to support healthcare[2]\u2014contributes to effective decision-making during diagnostic investigations by improving the ability to gather, organize, and display information, or by enhancing timely access to diagnostic reference information, as a result facilitating follow-up and providing valuable feedback to clinicians and patients.[3] Diagnostic informatics\u2014the use of digital health tools to facilitate the accuracy and timeliness of health information transfer and enhance the effectiveness of the decision-making processes[4]\u2014is fundamental for the safe and effective management of diagnostic investigations and the data that comes from them. Diagnostic informatics is thusly a key component of successful diagnostic stewardship, which encompasses the coordination of guidance and interventions to ensure the appropriate utilization of diagnostic tests for therapeutic decision-making.[5]\nThis paper will outline the importance of key diagnostic informatics and digital health stewardship initiatives across three interrelated areas: (1) diagnostic error and the establishment of outcome-based diagnostic research; (2) the safety and effectiveness of test result management and follow-up, including the critical role that patients can play in the process; and (3) digitally enhanced clinical decision support systems.\n\nDiagnostic error and the establishment of outcome-based diagnostic informatics research \nDiagnostic error is a major problem in healthcare, contributing to ~10% of patient deaths and between 6 to 17% of hospital adverse events.[6] Diagnostic error involves either the failure to establish an accurate and timely explanation of the patient's health problem(s) or the failure to effectively communicate that explanation to the patient.[6] In Australia, medico-legal data show that diagnostic error is implicated in half of medical negligence claims involving general practitioners.[7]\nMany factors can contribute to diagnostic error, including the lack of an integrated care pathway; the involvement of multiple specialists; problems with collaboration and communication among clinicians, patients, and their families; lack of infrastructure to support the diagnostic process; and inadequate attention to understanding the health problem and its causes.[6] Existing evidence shows that in acute care, an estimated 45% of laboratory testing is underutilized (i.e., when one or more tests should have been undertaken but weren't) and 21% is overutilized (i.e., when one or more tests were unnecessary or repeated within an inappropriate time frame).[8]\nThere is currently a lack of a systematic outcomes-based approach to identify and monitor the prevalence and impact of diagnostic errors. In part, this is because the typical relationship between a test and a health outcome is indirect. It is made harder still by the existence of data silos across different clinical and care settings (e.g., pathology, medical imaging, medical records, emergency, and hospital administration systems), which limit the ability to link test results and referrals to the different components of the patient journey (e.g., treatment and outcome). Despite the existence of evidence-based guidelines to encourage best practice utilization of diagnostic tests, there are very few studies on how these guidelines have impacted testing patterns and patient outcomes. This was noted by a recent international scoping review that drew attention to the significant lack of strategies for optimizing test utilization and improving patient outcomes in general practice.[9]\nLinked and integrated digital health data sources and repositories across hospital, pathology, and primary care provide a means to identify, measure, and monitor the quality of care. They can deliver key outcome-based measures of diagnostic utilization such as patient outcome or hospital admissions, which can be used to evaluate the impact of vital interventions (e.g., electronic decision support). This is the premise underlying critical initiatives like the New South Wales (NSW) Health Pathology Atlas of Variation (in collaboration with the NSW Emergency Care Institute and NSW eHealth Integrated Care), designed to create a statewide quality improvement project.[10]\nAs an example of the potential of outcome-focused studies to impact on patient care, we investigated the incidence of acute kidney injury (AKI) over a five-year period (2009\u20132013) across four NSW hospitals.[11] Patients with AKI were identified using the serum creatinine-based definition as described in the international consensus guidelines published by the \u201cKidney Disease: Improving Global Outcomes\u201d (KDIGO) Work Group. This required programming of a complex algorithm, which was run on more than two million creatinine results to detect the presence of AKI. The study found that only 15.9% of hospitalizations with AKI stage 1 were coded as such[11], thus highlighting the need for an evidence-based laboratory AKI decision support aid. Other examples of the value of linked data helping to power programs of research include measuring the effects of electronic medical records (EMRs) on healthcare[12][13], and the impact of the rapid flu tests on emergency department (ED) performance and patient outcomes.[14]\n\nEnhancing the safety and effectiveness of test result management and follow-up \nA significant source of diagnostic error can be attributed to shortcomings in the follow-up of test results, identified as a priority area by the World Health Organization's World Alliance for Patient Safety.[15] International evidence shows between 41 and 100% of patients leave the hospital with at least one test result pending at discharge, influenced in part by pressure on doctors to reduce length of stay.[16]\nCare transitions can be defined as the passage of patients among healthcare professionals, settings (e.g., units, wards, hospitals, aged care facilities), and their home.[17] Ineffective care transitions can lead to adverse events, such as medication errors[18], medical errors related to the completion of diagnostic workups[19], and the loss of crucial patient information.[17] Arriving at a diagnosis is not a single task but rather a series of tasks and responsibilities over numerous time points, shared among many people across the healthcare spectrum.[20]\nPathology and medical imaging services perform a major role in patient care by ensuring that reliable and accurate results are delivered in a timely fashion to inform clinical management. Nevertheless, there are large variations in how critically abnormal results are defined and reported, and the failure of laboratories to uniformly follow internationally recognized guidelines.[21] Several initiatives seek to address these issues, including The Royal College of Pathologists of Australasia (RCPA)\u2014Australasian Association of Clinical Biochemists (AACB) High Risk Results Working Party, which is developing a harmonized, evidence-based \u201cAlert List for Australasian Laboratories.\u201d[22] Similarly, the RCPA-led Pathology Information, Terminology and Units Standardizations (PITUS) program aims to establish standardized pathology information structures and terminologies to improve recording, decision support, and communication of laboratory information.[23][24]\nAnother initiaitve worth noting is a national Australian stakeholder forum (involving patient groups, clinicians, managers, and professional organizations), held in Sydney in 2016 to identify patient safety challenges related to shortcomings in the governance and integration of the test result management process and the underutilization of digital health solutions. The forum drew particular attention to the role patients can contribute to effective test result management. It recommended robust follow-up procedures particularly for tests that may be pending during transitions of care, critical test results, and non-critical but actionable test results, which can have a severe impact on patients' health outcomes.\nDrawing from these and other such examples, diagnostic informatics, along with diagnostic stewardship, provide a foundation stone for improved patient engagement and shared decision-making. This is because they incorporate shared information (enabling patients to read, comment on, and share in decisions about their care) and timely and meaningful communication (enabling patients to receive, send, and comprehend the information required).[25] By supporting patients to make informed decisions about their own health and the care they receive, this approach promotes person-centered care, an established key driver of quality in healthcare[26] (as depicted in Figure 1).\n\r\n\n\n\n\n\n\n\n\n\n\nFigure 1. Key features of person-centered care.\n\n\n\nDigitally enabled and enhanced decision support \nDigital health can contribute to effective decision-making by improving the ability to gather, organize, and display information, or by enhancing timely access to diagnostic reference information, facilitating follow-up and feedback to clinicians and patients. Electronic support can be used to[27]: \n\n optimize decision-making (e.g., choice of an appropriate laboratory test, improved compliance with care guidelines, improved interpretation of test results); \n improve care processes (e.g., better documentation and improved communication); and \n prevent errors of commission (e.g., unnecessary repetition of laboratory tests) and errors of omission (e.g., trigger immediate action in response to high-risk laboratory results).\nDigitally enabled decision support tools can be used in a wide variety of diagnostic-related areas, including as alerts and order sets (to enhance the appropriate choice of tests), as a provision of important information related to decision-making (to support the identification and management of risk factors), as guideline support, and as a provision of relevant reference material (to promote the best available care).\nThe Sensible Test Ordering Practice (STOP) initiative was developed in Australia to promote consistent and rational diagnostic test ordering practices in acute care settings. STOP involves the use of a traffic-light system (green, amber, or red) to restrict the range of tests that can be ordered depending on the seniority of the clinicians: \u201cgreen\u201d tests may be ordered by junior doctors, while more complex, expensive, or lower-yield tests designated \u201camber\u201d or \u201cred\u201d require senior medical staff sign off. The Australasian College for Emergency Medicine and the RCPA have developed a set of laboratory testing recommendations in EDs for common presentations based on the STOP principles.[28] An example for adults presenting to EDs with chest pain is presented in Table 1. The STOP guidelines can be integrated into electronic decision support tools linked to EMR systems to facilitate the consistency and appropriateness of diagnostic-test-ordering practices.\n\r\n\n\n\n\n\n\n\n\n\n\nTable 1. STOP principles for adults presenting to ED with chest pain\n\n\n\nWhile electronic decision support systems are often described as key to reducing misdiagnosis, their impact has not been strongly validated against patient outcomes, and their diffusion is patchy.[29] There is also a long history of failure associated with the use of computer-based tools.[29] These tools are often introduced to a clinical setting with limited pilot testing and little understanding of how they affect workflow.[29] Establishing effective digital health interventions must ensure the acceptance and usability of the digital health system and include systematic feedback about their performance to users.[3][30] The STOP electronic decision support initiative will thus need to be evaluated to ensure that it is effective (e.g., helps to improve the quality of care); efficient (e.g., is accessible and responsive to clinician and patient needs); appropriate (e.g., promotes the best possible outcomes of care); and safe (e.g., helps to optimize care quality and safety).\n\nDiscussion \nThe diagnostic process occurs over time and often involves multiple healthcare professionals across different care settings. Arriving at a diagnosis can be an uncertain process, often reliant on limited and imperfect data or indeterminate diagnostic interpretations from pathology reports.[6][31] The measurement of diagnosis is further challenged by traditional health record structures, which can fail to capture the crucial clinician and patient narrative, communication context, and organizational culture.[31] Recent advances in artificial intelligence and natural language processing tools (such as machine translation, speech recognition, and speech synthesis) provide fertile ground to improve and enhance our understanding of the diagnostic process.[32]\nTo ensure the seamless flow of, and access to, patient information, interoperable electronic health systems are critical.[6] However, a systematic review of studies of the impact of those digital health systems on test result management has shown that, in and of itself, digital health does not provide a complete solution.[33] The development of safe and effective digital health interventions needs to encompass multiple dimensions, taking into account patient-centered care (involving shared, timely, and meaningful information); existing diagnostic processes (involving multiple people across different clinical settings); and key organizational communication processes (multiple transactions requiring feedback, iteration, and confirmation).[33][34][35]\n\nAcknowledgements \nAuthor contributions \nAll authors contributed to the conception, design, and drafting of the work, drafting of the work, revising it critically for important intellectual content; provided final approval of the version to be published; and provided agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.\n\nFunding \nThe research described in this article was funded by a National Health and Medical Research Council Partnership Grant (APP1111925) involving Macquarie University, NSW Health Pathology, and the Australian Commission on Safety and Quality in Health Care.\n\nConflict of interest \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 Knottnerus, J.A.; van Weel, C.; Muris, J.W.M. (2002). \"Evaluation of diagnostic procedures\". BMJ 324 (7335): 477-80. doi:10.1136\/bmj.324.7335.477. PMC PMC1122397. PMID 11859054. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1122397 .   \n\n\u2191 Rowlands, D. (02 December 2019). \"What is digital health? And why does it matter?\" (PDF). Digital Health Workforce Academy. https:\/\/www.hisa.org.au\/wp-content\/uploads\/2019\/12\/What_is_Digital_Health.pdf?x97063 .   \n\n\u2191 3.0 3.1 El-Kareh, R.; Hasan, O.; Schiff, G.D. (2013). \"Use of health information technology to reduce diagnostic errors\". BMJ Quality and Safety 22 (Suppl 2): ii40-ii51. doi:10.1136\/bmjqs-2013-001884. PMC PMC3786650. PMID 23852973. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3786650 .   \n\n\u2191 Aller, R.D.; Georgiou, A.; Pantanowitz, L. (2012). \"Electronic Health Records\". In Pantanowitz, L.; Tuthill, J.M.; Balis, U.. Pathology Informatics: Theory and Practice. pp. 217\u201330. ISBN 9780891896197.   \n\n\u2191 Global Antimicrobial Resistance Surveillance System (2016). \"Diagnostic stewardship: A guide to implementation in antimicrobial resistance surveillance sites\". World Health Organization. https:\/\/www.who.int\/publications\/i\/item\/WHO-DGO-AMR-2016.3 .   \n\n\u2191 6.0 6.1 6.2 6.3 6.4 Balogh, E.P.; Miller, B.T.; Ball, J.R., ed. (2015). Improving Diagnosis in Health Care. The National Academies Press. doi:10.17226\/21794. ISBN 9780309377706.   \n\n\u2191 Bird, S. (23 February 2018). \"Reducing diagnostic error\". newsGP. https:\/\/www1.racgp.org.au\/newsgp\/professional\/reducing-diagnostic-error . Retrieved 18 January 2019 .   \n\n\u2191 Zhi, M.; Ding, E.L.; Theisen-Toupal, J. et al. (2013). \"The landscape of inappropriate laboratory testing: A 15-year meta-analysis\". PLoS One 8 (11): e78962. doi:10.1371\/journal.pone.0078962. PMC PMC3829815. PMID 24260139. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829815 .   \n\n\u2191 Elwenspoek, M.M.C.; Scott, L.J.; Alsop, K. et al. (2020). \"What methods are being used to create an evidence base on the use of laboratory tests to monitor long-term conditions in primary care? A scoping review\". Family Practice 37 (6): 845-853. doi:10.1093\/fampra\/cmaa074. PMC PMC7759753. PMID 32820328. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7759753 .   \n\n\u2191 Scowen, C.; Wabe, N.; Eigenstetter, A. et al. (2020). \"Evaluating the long-term effects of a data-driven approach to reduce variation in emergency department pathology investigations: study protocol for evaluation of the NSW Health Pathology Atlas of variation\". BMJ Open 10 (10): e039437. doi:10.1136\/bmjopen-2020-039437. PMC PMC7552857. PMID 33046472. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7552857 .   \n\n\u2191 11.0 11.1 Campbell, C.A.; Ling, L.; Kotwal, S. et al. (2020). \"Under-detection of acute kidney injury in hospitalised patients: A retrospective, multi-site, longitudinal study\". Internal Medicine Journal 50 (3): 307-314. doi:10.1111\/imj.14264. PMID 30816607.   \n\n\u2191 Georgiou, A.; Vecellio, E.; Toouli, G. et al. (2013). \"Monitoring the impact of the electronic medical record on the quality of laboratory test ordering practices\". Studies in Health Technologies and Informatics 188: 33\u20138. PMID 23823285.   \n\n\u2191 Vecellio, E.; Li, L.; Xiong, J. et al. (2015) (PDF). Examination of variation in hospital pathology investigations by Diagnosis-Related Groups and associations with outcomes and costs. Centre for Health Systems and Safety Research. ISBN 9781741384307. https:\/\/www1.health.gov.au\/internet\/main\/publishing.nsf\/Content\/0E3978777CE9637BCA2580C200088F78\/$File\/Final%20Report%20-%20Variation%20in%20Hospital%20Pathology%20-%20Macquarie%20University.pdf .   \n\n\u2191 Wabe, N.; Li, L.; Lindeman, R. et al. (2019). \"The impact of rapid molecular diagnostic testing for respiratory viruses on outcomes for emergency department patients\". Medical Journal of Australia 210 (7): 316\u201320. doi:10.5694\/mja2.50049. PMC PMC6617970. PMID 30838671. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6617970 .   \n\n\u2191 World Alliance for Patient Safety (2008). Jha, A.. ed. Summary of the evidence on patient safety: Implications for research. World Health Organization. ISBN 9789241596541. https:\/\/apps.who.int\/iris\/handle\/10665\/43874 .   \n\n\u2191 Callen, J.; Georgiou, A.; Li, J. et al. (2011). \"The safety implications of missed test results for hospitalised patients: A systematic review\". BMJ Quality and Safety 20 (2): 194-9. doi:10.1136\/bmjqs.2010.044339. PMC PMC3038104. PMID 21300992. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3038104 .   \n\n\u2191 17.0 17.1 National Quality Forum (September 2017). \"Improving Diagnostic Quality and Safety Final Report\". National Quality Forum. https:\/\/www.qualityforum.org\/Publications\/2017\/09\/Improving_Diagnostic_Quality_and_Safety_Final_Report.aspx . Retrieved 02 May 2018 .   \n\n\u2191 Kripalani, S.; Jackson, A.T.; Schnipper, J.L et al. (2007). \"Promoting effective transitions of care at hospital discharge: A review of key issues for hospitalists\". Journal of Hospital Medicine 2 (5): 314-23. doi:10.1002\/jhm.228. PMID 17935242.   \n\n\u2191 Moore, C.; Wisnivesky, J.; Williams, S. et al. (2003). \"Medical errors related to discontinuity of care from an inpatient to an outpatient setting\". Journal of General Internal Medicine 18 (8): 646-51. doi:10.1046\/j.1525-1497.2003.20722.x. PMC PMC1494907. PMID 12911647. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1494907 .   \n\n\u2191 Carayon, P.; Karsh, B.T.; Cartmill, R. et al. (2010). \"Incorporating health information technology into workflow redesign: Request for information summary report\". Agency for Healthcare Research and Quality. https:\/\/digital.ahrq.gov\/ahrq-funded-projects\/incorporating-health-information-technology-workflow-redesign\/citation\/workflow .   \n\n\u2191 Campbell, C.; Horvath, A. (2012). \"Towards harmonisation of critical laboratory result management - Review of the literature and survey of australasian practices\". Clinical Biochemist Reviews 33 (4): 149\u201360. PMC PMC3529552. PMID 23267247. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3529552 .   \n\n\u2191 Campbell, C.; Georgiou, A.; Westbrook, J.I. et al. (2016). \"What Alert Thresholds Should Be Used to Identify Critical Risk Results: A Systematic Review of the Evidence\". Clinical Chemistry 62 (11): 1445\u201357. doi:10.1373\/clinchem.2016.260638. PMID 27565161.   \n\n\u2191 Pathology Information, Terminology and Units Standardisation Program (July 2013). \"PITUS Update - Newsletter of the RCPA Pathology Information, Terminology and Units Standardisation Program\" (PDF). PITUS Update (Royal College of Pathologists of Australasia) (1). https:\/\/www.rcpa.edu.au\/Library\/Practising-Pathology\/PTIS\/PITUS\/Docs\/PITUS-Update-Issue-1-July-2013 .   \n\n\u2191 Hardie, R.-A.; Moore, D.; Holzhauser, D. et al. (2018). \"Informatics External Quality Assurance (IEQA) Down Under: evaluation of a pilot implementation\". Journal of Laboratory Medicine 42 (6): 297-304. doi:10.1515\/labmed-2018-0050.   \n\n\u2191 Joosten, E.A.G.; DeFuentes-Merillas, L.; de Weert, G.H. et al. (2008). \"Systematic review of the effects of shared decision-making on patient satisfaction, treatment adherence and health status\". Psychotherapy and Psychosomatics 77 (4): 219\u201326. doi:10.1159\/000126073. PMID 18418028.   \n\n\u2191 de Silva, D. (March 2014). \"Helping measure person-centred care: A review of evidence about commonly used approaches and tools used to help measure person-centred care\". The Health Foundation. https:\/\/www.health.org.uk\/publications\/helping-measure-person-centred-care .   \n\n\u2191 Osheroff, J.; Pifer, E.; Teich, J. et al. (2005). Improving Outcomes with Clinical Decision Support: An Implementer's Guide (1st ed.). Productivity Press. ISBN 9780976127727.   \n\n\u2191 Australasian College for Emergency Medicine, Royal College of Pathologists of Australasia (December 2018). \"Pathology Testing in the Emergency Department\" (PDF). Royal College of Pathologists of Australasia. https:\/\/www.rcpa.edu.au\/Library\/College-Policies\/Guidelines\/Pathology-Testing-in-the-Emergency-Department.aspx .   \n\n\u2191 29.0 29.1 29.2 Garg, A.X.; Adhikari, N.K.J.; McDonald, H. et al. (2005). \"Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: A systematic review\". JAMA 293 (10): 1223-38. doi:10.1001\/jama.293.10.1223. PMID 15755945.   \n\n\u2191 Georgiou, A.; Prgomet, M. (2019). \"E-health: Developing trust, confidence, quality and sustainability\". Health Information Management 48 (2): 59\u201361. doi:10.1177\/1833358318808064. PMID 30392402.   \n\n\u2191 31.0 31.1 Miao, M.; Dahm, M.R.; Li, J. et al. (2020). \"Managing Uncertainty During the Communication of Diagnostic Test Information Between Patients and Clinicians in Australian Emergency Care\". Qualitative Health Research 30 (8): 1287-1300. doi:10.1177\/1049732320913037. PMID 32249721.   \n\n\u2191 Hirschberg, J.; Manning, C.D. (2015). \"Advances in natural language processing\". Science 349 (6245): 261\u20136. doi:10.1126\/science.aaa8685. PMID 26185244.   \n\n\u2191 33.0 33.1 Georgiou, A.; Li, J.; Thomas, J. et al. (2019). \"The impact of health information technology on the management and follow-up of test results - A systematic review\". JAMIA 26 (7): 678\u2013688. doi:10.1093\/jamia\/ocz032. PMC PMC6562156. PMID 31192362. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6562156 .   \n\n\u2191 Georgiou, A.; Westbrook, J.I.; Braithwaite, J. (2011). \"Time matters: A theoretical and empirical examination of the temporal landscape of a hospital pathology service and the impact of e-health\". Social Science and Medicine 72 (10): 1603-10. doi:10.1016\/j.socscimed.2011.03.020. PMID 21497430.   \n\n\u2191 Georgiou, A.; Westbrook, J.I.; Braithwaite, J. (2012). \"An empirically-derived approach for investigating Health Information Technology: The Elementally Entangled Organisational Communication (EEOC) framework\". BMC Medical Informatics and Decision Making 12: 68. doi:10.1186\/1472-6947-12-68. PMC PMC3407796. PMID 22788698. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3407796 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\">https:\/\/www.limswiki.org\/index.php\/Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2021)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on pathology informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \n\t\n\t\n\t\r\n\n\t\r\n\n \n\t\n\t\r\n\n\t\r\n\n\t\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 29 June 2021, at 16:14.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 51 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n","236fa37968f1db59b5299967fce1aa76_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Diagnostic_informatics_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence_safety_and_value skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Diagnostic informatics: The role of digital health in diagnostic stewardship and the achievement of excellence, safety, and value<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Diagnostic investigations (i.e., pathology <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> analysis and <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_imaging\" title=\"Medical imaging\" class=\"wiki-link\" data-key=\"dddd7e2b5706415d7af0375386e6eafa\">medical imaging<\/a>) aim to increase the certainty of the presence of or absence of disease by supporting the process of differential diagnosis, support clinical management, and monitor a patient's trajectory (e.g., disease progression or response to treatment). Digital health\u2014defined as the collection, storage, retrieval, transmission, and utilization of data, <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, and knowledge to support healthcare\u2014has become an essential component of the diagnostic investigational process, helping to facilitate the accuracy and timeliness of information transfer and enhance the effectiveness of decision-making processes. Digital health is also important to diagnostic stewardship, which involves coordinated guidance and interventions to ensure the appropriate utilization of <a href=\"https:\/\/www.limswiki.org\/index.php\/Diagnostic_laboratory\" title=\"Diagnostic laboratory\" class=\"mw-redirect wiki-link\" data-key=\"4dea9b8dbc15ed897d7aff1b35c381ba\">diagnostic tests<\/a> for therapeutic decision-making. Diagnostic stewardship and <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics_(academic_field)\" title=\"Informatics (academic field)\" class=\"wiki-link\" data-key=\"0391318826a5d9f9a1a1bcc88394739f\">informatics<\/a> are thus important in efforts to establish shared decision-making. This is because they contribute to the establishment of shared information platforms (which enable patients to read, comment on, and share in decisions about their care) based on timely and meaningful communication. \n<\/p><p>This paper will outline key diagnostic informatics and stewardship initiatives across three interrelated fields: (1) diagnostic error and the establishment of outcome-based diagnostic research, (2) the safety and effectiveness of test result management and follow-up, and (3) digitally enhanced <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support systems<\/a>.\n<\/p><p><b>Keywords<\/b>: pathology, diagnostic error, outcomes based assessment, evaluation, safety, health informatics, decision support, test result follow-up\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Diagnostic investigations and tests (e.g., pathology and <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_imaging\" title=\"Medical imaging\" class=\"wiki-link\" data-key=\"dddd7e2b5706415d7af0375386e6eafa\">medical imaging<\/a>) involve the observation of personal characteristics, symptoms, signs, and history. Their aim is to increase certainty of the presence of or absence of disease (or in the case of the differential diagnosis process, distinguish between different possible diagnoses), support clinical management, and monitor a patient's trajectory (e.g., during or after treatment).<sup id=\"rdp-ebb-cite_ref-KnottnerusEval02_1-0\" class=\"reference\"><a href=\"#cite_note-KnottnerusEval02-1\">[1]<\/a><\/sup> Most recently, <a href=\"https:\/\/www.limswiki.org\/index.php\/Diagnostic_laboratory\" title=\"Diagnostic laboratory\" class=\"mw-redirect wiki-link\" data-key=\"4dea9b8dbc15ed897d7aff1b35c381ba\">diagnostic investigations<\/a> have had a pivotal role in the response to the <a href=\"https:\/\/www.limswiki.org\/index.php\/SARS-CoV-2\" title=\"SARS-CoV-2\" class=\"mw-redirect wiki-link\" data-key=\"6c6b78479de2c640028696ed32948526\">SARS-CoV-2<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/Pandemic\" title=\"Pandemic\" class=\"wiki-link\" data-key=\"bd9a48e6c6e41b6d603ee703836b01f1\">pandemic<\/a> through <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> testing and <a href=\"https:\/\/www.limswiki.org\/index.php\/Epidemiology\" title=\"Epidemiology\" class=\"wiki-link\" data-key=\"123badb8bf0b37a513182dbcfc3875bc\">epidemiological<\/a> surveillance of viral infection, not only from its detection and diagnosis (e.g., via <a href=\"https:\/\/www.limswiki.org\/index.php\/Molecular_diagnostics\" title=\"Molecular diagnostics\" class=\"wiki-link\" data-key=\"8fc14cae7a6fbac9a53fae1394fae7ee\">molecular testing<\/a> by <a href=\"https:\/\/www.limswiki.org\/index.php\/Polymerase_chain_reaction\" title=\"Polymerase chain reaction\" class=\"wiki-link\" data-key=\"f6569fb01ef396379f9f4efa4527e715\">real-time polymerase chain reaction<\/a> [RT-PCR]) but also its prognostication, treatment, monitoring (e.g., testing related to comorbidities), and recovery (aided by anti-SARS-CoV-2 monoclonal antibodies).\n<\/p><p>Digital health\u2014defined as the collection, storage, retrieval, transmission, and utilization of data, <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, and knowledge to support healthcare<sup id=\"rdp-ebb-cite_ref-RowlandsWhat19_2-0\" class=\"reference\"><a href=\"#cite_note-RowlandsWhat19-2\">[2]<\/a><\/sup>\u2014contributes to effective decision-making during diagnostic investigations by improving the ability to gather, organize, and display information, or by enhancing timely access to diagnostic reference information, as a result facilitating follow-up and providing valuable feedback to clinicians and patients.<sup id=\"rdp-ebb-cite_ref-El-KarehUse13_3-0\" class=\"reference\"><a href=\"#cite_note-El-KarehUse13-3\">[3]<\/a><\/sup> <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">Diagnostic informatics<\/a>\u2014the use of digital health tools to facilitate the accuracy and timeliness of health information transfer and enhance the effectiveness of the decision-making processes<sup id=\"rdp-ebb-cite_ref-AllerElect12_4-0\" class=\"reference\"><a href=\"#cite_note-AllerElect12-4\">[4]<\/a><\/sup>\u2014is fundamental for the safe and effective management of diagnostic investigations and the data that comes from them. Diagnostic informatics is thusly a key component of successful diagnostic stewardship, which encompasses the coordination of guidance and interventions to ensure the appropriate utilization of diagnostic tests for therapeutic decision-making.<sup id=\"rdp-ebb-cite_ref-WHODiag16_5-0\" class=\"reference\"><a href=\"#cite_note-WHODiag16-5\">[5]<\/a><\/sup>\n<\/p><p>This paper will outline the importance of key diagnostic informatics and digital health stewardship initiatives across three interrelated areas: (1) diagnostic error and the establishment of outcome-based diagnostic research; (2) the safety and effectiveness of test result management and follow-up, including the critical role that patients can play in the process; and (3) digitally enhanced <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support systems<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Diagnostic_error_and_the_establishment_of_outcome-based_diagnostic_informatics_research\">Diagnostic error and the establishment of outcome-based diagnostic informatics research<\/span><\/h2>\n<p>Diagnostic error is a major problem in healthcare, contributing to ~10% of patient deaths and between 6 to 17% of <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> adverse events.<sup id=\"rdp-ebb-cite_ref-BaloghImprov15_6-0\" class=\"reference\"><a href=\"#cite_note-BaloghImprov15-6\">[6]<\/a><\/sup> Diagnostic error involves either the failure to establish an accurate and timely explanation of the patient's health problem(s) or the failure to effectively communicate that explanation to the patient.<sup id=\"rdp-ebb-cite_ref-BaloghImprov15_6-1\" class=\"reference\"><a href=\"#cite_note-BaloghImprov15-6\">[6]<\/a><\/sup> In Australia, medico-legal data show that diagnostic error is implicated in half of medical negligence claims involving general practitioners.<sup id=\"rdp-ebb-cite_ref-BirdReduc18_7-0\" class=\"reference\"><a href=\"#cite_note-BirdReduc18-7\">[7]<\/a><\/sup>\n<\/p><p>Many factors can contribute to diagnostic error, including the lack of an integrated care pathway; the involvement of multiple specialists; problems with collaboration and communication among clinicians, patients, and their families; lack of infrastructure to support the diagnostic process; and inadequate attention to understanding the health problem and its causes.<sup id=\"rdp-ebb-cite_ref-BaloghImprov15_6-2\" class=\"reference\"><a href=\"#cite_note-BaloghImprov15-6\">[6]<\/a><\/sup> Existing evidence shows that in acute care, an estimated 45% of laboratory testing is underutilized (i.e., when one or more tests should have been undertaken but weren't) and 21% is overutilized (i.e., when one or more tests were unnecessary or repeated within an inappropriate time frame).<sup id=\"rdp-ebb-cite_ref-ZhiTheLand13_8-0\" class=\"reference\"><a href=\"#cite_note-ZhiTheLand13-8\">[8]<\/a><\/sup>\n<\/p><p>There is currently a lack of a systematic outcomes-based approach to identify and monitor the prevalence and impact of diagnostic errors. In part, this is because the typical relationship between a test and a health outcome is indirect. It is made harder still by the existence of data silos across different clinical and care settings (e.g., pathology, medical imaging, medical records, emergency, and hospital administration systems), which limit the ability to link test results and referrals to the different components of the patient journey (e.g., treatment and outcome). Despite the existence of evidence-based guidelines to encourage best practice utilization of diagnostic tests, there are very few studies on how these guidelines have impacted testing patterns and patient outcomes. This was noted by a recent international scoping review that drew attention to the significant lack of strategies for optimizing test utilization and improving patient outcomes in general practice.<sup id=\"rdp-ebb-cite_ref-ElwenspoekWhat20_9-0\" class=\"reference\"><a href=\"#cite_note-ElwenspoekWhat20-9\">[9]<\/a><\/sup>\n<\/p><p>Linked and integrated digital health data sources and repositories across hospital, pathology, and primary care provide a means to identify, measure, and monitor the quality of care. They can deliver key outcome-based measures of diagnostic utilization such as patient outcome or hospital admissions, which can be used to evaluate the impact of vital interventions (e.g., electronic decision support). This is the premise underlying critical initiatives like the New South Wales (NSW) Health Pathology Atlas of Variation (in collaboration with the NSW Emergency Care Institute and NSW eHealth Integrated Care), designed to create a statewide quality improvement project.<sup id=\"rdp-ebb-cite_ref-ScowenEval20_10-0\" class=\"reference\"><a href=\"#cite_note-ScowenEval20-10\">[10]<\/a><\/sup>\n<\/p><p>As an example of the potential of outcome-focused studies to impact on patient care, we investigated the incidence of acute kidney injury (AKI) over a five-year period (2009\u20132013) across four NSW hospitals.<sup id=\"rdp-ebb-cite_ref-CampbellUnder20_11-0\" class=\"reference\"><a href=\"#cite_note-CampbellUnder20-11\">[11]<\/a><\/sup> Patients with AKI were identified using the serum creatinine-based definition as described in the international consensus guidelines published by the \u201cKidney Disease: Improving Global Outcomes\u201d (KDIGO) Work Group. This required programming of a complex algorithm, which was run on more than two million creatinine results to detect the presence of AKI. The study found that only 15.9% of hospitalizations with AKI stage 1 were coded as such<sup id=\"rdp-ebb-cite_ref-CampbellUnder20_11-1\" class=\"reference\"><a href=\"#cite_note-CampbellUnder20-11\">[11]<\/a><\/sup>, thus highlighting the need for an evidence-based laboratory AKI decision support aid. Other examples of the value of linked data helping to power programs of research include measuring the effects of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical records<\/a> (EMRs) on healthcare<sup id=\"rdp-ebb-cite_ref-GeorgiouMonit13_12-0\" class=\"reference\"><a href=\"#cite_note-GeorgiouMonit13-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VecellioExam15_13-0\" class=\"reference\"><a href=\"#cite_note-VecellioExam15-13\">[13]<\/a><\/sup>, and the impact of the rapid flu tests on emergency department (ED) performance and patient outcomes.<sup id=\"rdp-ebb-cite_ref-WabeTheImp19_14-0\" class=\"reference\"><a href=\"#cite_note-WabeTheImp19-14\">[14]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Enhancing_the_safety_and_effectiveness_of_test_result_management_and_follow-up\">Enhancing the safety and effectiveness of test result management and follow-up<\/span><\/h2>\n<p>A significant source of diagnostic error can be attributed to shortcomings in the follow-up of test results, identified as a priority area by the <a href=\"https:\/\/www.limswiki.org\/index.php\/World_Health_Organization\" title=\"World Health Organization\" class=\"wiki-link\" data-key=\"2a49f1470638d5f579e3f6419e239b03\">World Health Organization<\/a>'s World Alliance for Patient Safety.<sup id=\"rdp-ebb-cite_ref-JhaSumm08_15-0\" class=\"reference\"><a href=\"#cite_note-JhaSumm08-15\">[15]<\/a><\/sup> International evidence shows between 41 and 100% of patients leave the hospital with at least one test result pending at discharge, influenced in part by pressure on doctors to reduce length of stay.<sup id=\"rdp-ebb-cite_ref-CallenTheSafe11_16-0\" class=\"reference\"><a href=\"#cite_note-CallenTheSafe11-16\">[16]<\/a><\/sup>\n<\/p><p>Care transitions can be defined as the passage of patients among healthcare professionals, settings (e.g., units, wards, hospitals, aged care facilities), and their home.<sup id=\"rdp-ebb-cite_ref-NQFImprov17_17-0\" class=\"reference\"><a href=\"#cite_note-NQFImprov17-17\">[17]<\/a><\/sup> Ineffective care transitions can lead to adverse events, such as medication errors<sup id=\"rdp-ebb-cite_ref-KripalaniPromo07_18-0\" class=\"reference\"><a href=\"#cite_note-KripalaniPromo07-18\">[18]<\/a><\/sup>, medical errors related to the completion of diagnostic workups<sup id=\"rdp-ebb-cite_ref-MooreMed03_19-0\" class=\"reference\"><a href=\"#cite_note-MooreMed03-19\">[19]<\/a><\/sup>, and the loss of crucial patient information.<sup id=\"rdp-ebb-cite_ref-NQFImprov17_17-1\" class=\"reference\"><a href=\"#cite_note-NQFImprov17-17\">[17]<\/a><\/sup> Arriving at a diagnosis is not a single task but rather a series of tasks and responsibilities over numerous time points, shared among many people across the healthcare spectrum.<sup id=\"rdp-ebb-cite_ref-CarayonIncorp10_20-0\" class=\"reference\"><a href=\"#cite_note-CarayonIncorp10-20\">[20]<\/a><\/sup>\n<\/p><p>Pathology and medical imaging services perform a major role in patient care by ensuring that reliable and accurate results are delivered in a timely fashion to inform clinical management. Nevertheless, there are large variations in how critically abnormal results are defined and reported, and the failure of laboratories to uniformly follow internationally recognized guidelines.<sup id=\"rdp-ebb-cite_ref-CampbellTowards12_21-0\" class=\"reference\"><a href=\"#cite_note-CampbellTowards12-21\">[21]<\/a><\/sup> Several initiatives seek to address these issues, including The Royal College of Pathologists of Australasia (RCPA)\u2014Australasian Association of Clinical Biochemists (AACB) High Risk Results Working Party, which is developing a harmonized, evidence-based \u201cAlert List for Australasian Laboratories.\u201d<sup id=\"rdp-ebb-cite_ref-CampbellWhat16_22-0\" class=\"reference\"><a href=\"#cite_note-CampbellWhat16-22\">[22]<\/a><\/sup> Similarly, the RCPA-led Pathology Information, Terminology and Units Standardizations (PITUS) program aims to establish standardized pathology information structures and terminologies to improve recording, decision support, and communication of laboratory information.<sup id=\"rdp-ebb-cite_ref-PITUSJuly13_23-0\" class=\"reference\"><a href=\"#cite_note-PITUSJuly13-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HardieInformatics18_24-0\" class=\"reference\"><a href=\"#cite_note-HardieInformatics18-24\">[24]<\/a><\/sup>\n<\/p><p>Another initiaitve worth noting is a national Australian stakeholder forum (involving patient groups, clinicians, managers, and professional organizations), held in Sydney in 2016 to identify patient safety challenges related to shortcomings in the governance and integration of the test result management process and the underutilization of digital health solutions. The forum drew particular attention to the role patients can contribute to effective test result management. It recommended robust follow-up procedures particularly for tests that may be pending during transitions of care, critical test results, and non-critical but actionable test results, which can have a severe impact on patients' health outcomes.\n<\/p><p>Drawing from these and other such examples, diagnostic informatics, along with diagnostic stewardship, provide a foundation stone for improved patient engagement and shared decision-making. This is because they incorporate shared information (enabling patients to read, comment on, and share in decisions about their care) and timely and meaningful communication (enabling patients to receive, send, and comprehend the information required).<sup id=\"rdp-ebb-cite_ref-JoostenSys08_25-0\" class=\"reference\"><a href=\"#cite_note-JoostenSys08-25\">[25]<\/a><\/sup> By supporting patients to make informed decisions about their own health and the care they receive, this approach promotes person-centered care, an established key driver of quality in healthcare<sup id=\"rdp-ebb-cite_ref-deSilvaHelp14_26-0\" class=\"reference\"><a href=\"#cite_note-deSilvaHelp14-26\">[26]<\/a><\/sup> (as depicted in Figure 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Georgiou_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"b1ef7be001875ba505ebbc8ee27f527f\"><img alt=\"Fig1 Georgiou FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/1f\/Fig1_Georgiou_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 1.<\/b> Key features of person-centered care.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Digitally_enabled_and_enhanced_decision_support\">Digitally enabled and enhanced decision support<\/span><\/h2>\n<p>Digital health can contribute to effective decision-making by improving the ability to gather, organize, and display information, or by enhancing timely access to diagnostic reference information, facilitating follow-up and feedback to clinicians and patients. Electronic support can be used to<sup id=\"rdp-ebb-cite_ref-OsheroffImprov05_27-0\" class=\"reference\"><a href=\"#cite_note-OsheroffImprov05-27\">[27]<\/a><\/sup>: \n<\/p>\n<ul><li> optimize decision-making (e.g., choice of an appropriate laboratory test, improved compliance with care guidelines, improved interpretation of test results); <\/li>\n<li> improve care processes (e.g., better documentation and improved communication); and <\/li>\n<li> prevent errors of commission (e.g., unnecessary repetition of laboratory tests) and errors of omission (e.g., trigger immediate action in response to high-risk laboratory results).<\/li><\/ul>\n<p>Digitally enabled decision support tools can be used in a wide variety of diagnostic-related areas, including as alerts and order sets (to enhance the appropriate choice of tests), as a provision of important information related to decision-making (to support the identification and management of risk factors), as guideline support, and as a provision of relevant reference material (to promote the best available care).\n<\/p><p>The Sensible Test Ordering Practice (STOP) initiative was developed in Australia to promote consistent and rational diagnostic test ordering practices in acute care settings. STOP involves the use of a traffic-light system (green, amber, or red) to restrict the range of tests that can be ordered depending on the seniority of the clinicians: \u201cgreen\u201d tests may be ordered by junior doctors, while more complex, expensive, or lower-yield tests designated \u201camber\u201d or \u201cred\u201d require senior medical staff sign off. The Australasian College for Emergency Medicine and the RCPA have developed a set of laboratory testing recommendations in EDs for common presentations based on the STOP principles.<sup id=\"rdp-ebb-cite_ref-RCPAPath13_28-0\" class=\"reference\"><a href=\"#cite_note-RCPAPath13-28\">[28]<\/a><\/sup> An example for adults presenting to EDs with chest pain is presented in Table 1. The STOP guidelines can be integrated into electronic decision support tools linked to EMR systems to facilitate the consistency and appropriateness of diagnostic-test-ordering practices.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Georgiou_FrontDigHlth2021_3.jpg\" class=\"image wiki-link\" data-key=\"f3e2420e86333246e191fa7e0c1f17da\"><img alt=\"Tab1 Georgiou FrontDigHlth2021 3.jpg\" src=\"https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/94\/Tab1_Georgiou_FrontDigHlth2021_3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Table 1.<\/b> STOP principles for adults presenting to ED with chest pain<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>While electronic decision support systems are often described as key to reducing misdiagnosis, their impact has not been strongly validated against patient outcomes, and their diffusion is patchy.<sup id=\"rdp-ebb-cite_ref-GargEffects05_29-0\" class=\"reference\"><a href=\"#cite_note-GargEffects05-29\">[29]<\/a><\/sup> There is also a long history of failure associated with the use of computer-based tools.<sup id=\"rdp-ebb-cite_ref-GargEffects05_29-1\" class=\"reference\"><a href=\"#cite_note-GargEffects05-29\">[29]<\/a><\/sup> These tools are often introduced to a clinical setting with limited pilot testing and little understanding of how they affect workflow.<sup id=\"rdp-ebb-cite_ref-GargEffects05_29-2\" class=\"reference\"><a href=\"#cite_note-GargEffects05-29\">[29]<\/a><\/sup> Establishing effective digital health interventions must ensure the acceptance and usability of the digital health system and include systematic feedback about their performance to users.<sup id=\"rdp-ebb-cite_ref-El-KarehUse13_3-1\" class=\"reference\"><a href=\"#cite_note-El-KarehUse13-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GeorgiouEhealth19_30-0\" class=\"reference\"><a href=\"#cite_note-GeorgiouEhealth19-30\">[30]<\/a><\/sup> The STOP electronic decision support initiative will thus need to be evaluated to ensure that it is effective (e.g., helps to improve the quality of care); efficient (e.g., is accessible and responsive to clinician and patient needs); appropriate (e.g., promotes the best possible outcomes of care); and safe (e.g., helps to optimize care quality and safety).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>The diagnostic process occurs over time and often involves multiple healthcare professionals across different care settings. Arriving at a diagnosis can be an uncertain process, often reliant on limited and imperfect data or indeterminate diagnostic interpretations from pathology reports.<sup id=\"rdp-ebb-cite_ref-BaloghImprov15_6-3\" class=\"reference\"><a href=\"#cite_note-BaloghImprov15-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MiaoManag20_31-0\" class=\"reference\"><a href=\"#cite_note-MiaoManag20-31\">[31]<\/a><\/sup> The measurement of diagnosis is further challenged by traditional health record structures, which can fail to capture the crucial clinician and patient narrative, communication context, and organizational culture.<sup id=\"rdp-ebb-cite_ref-MiaoManag20_31-1\" class=\"reference\"><a href=\"#cite_note-MiaoManag20-31\">[31]<\/a><\/sup> Recent advances in <a href=\"https:\/\/www.limswiki.org\/index.php\/Artificial_intelligence\" title=\"Artificial intelligence\" class=\"wiki-link\" data-key=\"0c45a597361ca47e1cd8112af676276e\">artificial intelligence<\/a> and natural language processing tools (such as machine translation, speech recognition, and speech synthesis) provide fertile ground to improve and enhance our understanding of the diagnostic process.<sup id=\"rdp-ebb-cite_ref-HirschbergAdvan15_32-0\" class=\"reference\"><a href=\"#cite_note-HirschbergAdvan15-32\">[32]<\/a><\/sup>\n<\/p><p>To ensure the seamless flow of, and access to, patient information, interoperable electronic health systems are critical.<sup id=\"rdp-ebb-cite_ref-BaloghImprov15_6-4\" class=\"reference\"><a href=\"#cite_note-BaloghImprov15-6\">[6]<\/a><\/sup> However, a systematic review of studies of the impact of those digital health systems on test result management has shown that, in and of itself, digital health does not provide a complete solution.<sup id=\"rdp-ebb-cite_ref-GeorgiouTheImp19_33-0\" class=\"reference\"><a href=\"#cite_note-GeorgiouTheImp19-33\">[33]<\/a><\/sup> The development of safe and effective digital health interventions needs to encompass multiple dimensions, taking into account patient-centered care (involving shared, timely, and meaningful information); existing diagnostic processes (involving multiple people across different clinical settings); and key organizational communication processes (multiple transactions requiring feedback, iteration, and confirmation).<sup id=\"rdp-ebb-cite_ref-GeorgiouTheImp19_33-1\" class=\"reference\"><a href=\"#cite_note-GeorgiouTheImp19-33\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GeorgiouTime11_34-0\" class=\"reference\"><a href=\"#cite_note-GeorgiouTime11-34\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GeorgiouAnEmp12_35-0\" class=\"reference\"><a href=\"#cite_note-GeorgiouAnEmp12-35\">[35]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>All authors contributed to the conception, design, and drafting of the work, drafting of the work, revising it critically for important intellectual content; provided final approval of the version to be published; and provided agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>The research described in this article was funded by a National Health and Medical Research Council Partnership Grant (APP1111925) involving Macquarie University, NSW Health Pathology, and the Australian Commission on Safety and Quality in Health Care.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-KnottnerusEval02-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KnottnerusEval02_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Knottnerus, J.A.; van Weel, C.; Muris, J.W.M. (2002). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1122397\" target=\"_blank\">\"Evaluation of diagnostic procedures\"<\/a>. <i>BMJ<\/i> <b>324<\/b> (7335): 477-80. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.324.7335.477\" target=\"_blank\">10.1136\/bmj.324.7335.477<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1122397\/\" target=\"_blank\">PMC1122397<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11859054\" target=\"_blank\">11859054<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1122397\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1122397<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+of+diagnostic+procedures&rft.jtitle=BMJ&rft.aulast=Knottnerus%2C+J.A.%3B+van+Weel%2C+C.%3B+Muris%2C+J.W.M.&rft.au=Knottnerus%2C+J.A.%3B+van+Weel%2C+C.%3B+Muris%2C+J.W.M.&rft.date=2002&rft.volume=324&rft.issue=7335&rft.pages=477-80&rft_id=info:doi\/10.1136%2Fbmj.324.7335.477&rft_id=info:pmc\/PMC1122397&rft_id=info:pmid\/11859054&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1122397&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RowlandsWhat19-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RowlandsWhat19_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Rowlands, D. (02 December 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.hisa.org.au\/wp-content\/uploads\/2019\/12\/What_is_Digital_Health.pdf?x97063\" target=\"_blank\">\"What is digital health? And why does it matter?\"<\/a> (PDF). Digital Health Workforce Academy<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.hisa.org.au\/wp-content\/uploads\/2019\/12\/What_is_Digital_Health.pdf?x97063\" target=\"_blank\">https:\/\/www.hisa.org.au\/wp-content\/uploads\/2019\/12\/What_is_Digital_Health.pdf?x97063<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+is+digital+health%3F+And+why+does+it+matter%3F&rft.atitle=&rft.aulast=Rowlands%2C+D.&rft.au=Rowlands%2C+D.&rft.date=02+December+2019&rft.pub=Digital+Health+Workforce+Academy&rft_id=https%3A%2F%2Fwww.hisa.org.au%2Fwp-content%2Fuploads%2F2019%2F12%2FWhat_is_Digital_Health.pdf%3Fx97063&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-El-KarehUse13-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-El-KarehUse13_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-El-KarehUse13_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">El-Kareh, R.; Hasan, O.; Schiff, G.D. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3786650\" target=\"_blank\">\"Use of health information technology to reduce diagnostic errors\"<\/a>. <i>BMJ Quality and Safety<\/i> <b>22<\/b> (Suppl 2): ii40-ii51. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmjqs-2013-001884\" target=\"_blank\">10.1136\/bmjqs-2013-001884<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3786650\/\" target=\"_blank\">PMC3786650<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23852973\" target=\"_blank\">23852973<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3786650\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3786650<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Use+of+health+information+technology+to+reduce+diagnostic+errors&rft.jtitle=BMJ+Quality+and+Safety&rft.aulast=El-Kareh%2C+R.%3B+Hasan%2C+O.%3B+Schiff%2C+G.D.&rft.au=El-Kareh%2C+R.%3B+Hasan%2C+O.%3B+Schiff%2C+G.D.&rft.date=2013&rft.volume=22&rft.issue=Suppl+2&rft.pages=ii40-ii51&rft_id=info:doi\/10.1136%2Fbmjqs-2013-001884&rft_id=info:pmc\/PMC3786650&rft_id=info:pmid\/23852973&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3786650&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AllerElect12-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AllerElect12_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Aller, R.D.; Georgiou, A.; Pantanowitz, L. (2012). \"Electronic Health Records\". In Pantanowitz, L.; Tuthill, J.M.; Balis, U.. <i>Pathology Informatics: Theory and Practice<\/i>. pp. 217\u201330. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780891896197.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Electronic+Health+Records&rft.atitle=Pathology+Informatics%3A+Theory+and+Practice&rft.aulast=Aller%2C+R.D.%3B+Georgiou%2C+A.%3B+Pantanowitz%2C+L.&rft.au=Aller%2C+R.D.%3B+Georgiou%2C+A.%3B+Pantanowitz%2C+L.&rft.date=2012&rft.pages=pp.%26nbsp%3B217%E2%80%9330&rft.isbn=9780891896197&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WHODiag16-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WHODiag16_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Global Antimicrobial Resistance Surveillance System (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.who.int\/publications\/i\/item\/WHO-DGO-AMR-2016.3\" target=\"_blank\">\"Diagnostic stewardship: A guide to implementation in antimicrobial resistance surveillance sites\"<\/a>. World Health Organization<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.who.int\/publications\/i\/item\/WHO-DGO-AMR-2016.3\" target=\"_blank\">https:\/\/www.who.int\/publications\/i\/item\/WHO-DGO-AMR-2016.3<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Diagnostic+stewardship%3A+A+guide+to+implementation+in+antimicrobial+resistance+surveillance+sites&rft.atitle=&rft.aulast=Global+Antimicrobial+Resistance+Surveillance+System&rft.au=Global+Antimicrobial+Resistance+Surveillance+System&rft.date=2016&rft.pub=World+Health+Organization&rft_id=https%3A%2F%2Fwww.who.int%2Fpublications%2Fi%2Fitem%2FWHO-DGO-AMR-2016.3&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaloghImprov15-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BaloghImprov15_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-BaloghImprov15_6-1\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-BaloghImprov15_6-2\">6.2<\/a><\/sup> <sup><a href=\"#cite_ref-BaloghImprov15_6-3\">6.3<\/a><\/sup> <sup><a href=\"#cite_ref-BaloghImprov15_6-4\">6.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Balogh, E.P.; Miller, B.T.; Ball, J.R., ed. (2015). <i>Improving Diagnosis in Health Care<\/i>. The National Academies Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.17226%2F21794\" target=\"_blank\">10.17226\/21794<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780309377706.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Improving+Diagnosis+in+Health+Care&rft.date=2015&rft.pub=The+National+Academies+Press&rft_id=info:doi\/10.17226%2F21794&rft.isbn=9780309377706&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BirdReduc18-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BirdReduc18_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bird, S. (23 February 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www1.racgp.org.au\/newsgp\/professional\/reducing-diagnostic-error\" target=\"_blank\">\"Reducing diagnostic error\"<\/a>. <i>newsGP<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www1.racgp.org.au\/newsgp\/professional\/reducing-diagnostic-error\" target=\"_blank\">https:\/\/www1.racgp.org.au\/newsgp\/professional\/reducing-diagnostic-error<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Reducing+diagnostic+error&rft.atitle=newsGP&rft.aulast=Bird%2C+S.&rft.au=Bird%2C+S.&rft.date=23+February+2018&rft_id=https%3A%2F%2Fwww1.racgp.org.au%2Fnewsgp%2Fprofessional%2Freducing-diagnostic-error&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhiTheLand13-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhiTheLand13_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhi, M.; Ding, E.L.; Theisen-Toupal, J. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829815\" target=\"_blank\">\"The landscape of inappropriate laboratory testing: A 15-year meta-analysis\"<\/a>. <i>PLoS One<\/i> <b>8<\/b> (11): e78962. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0078962\" target=\"_blank\">10.1371\/journal.pone.0078962<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3829815\/\" target=\"_blank\">PMC3829815<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24260139\" target=\"_blank\">24260139<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829815\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829815<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+landscape+of+inappropriate+laboratory+testing%3A+A+15-year+meta-analysis&rft.jtitle=PLoS+One&rft.aulast=Zhi%2C+M.%3B+Ding%2C+E.L.%3B+Theisen-Toupal%2C+J.+et+al.&rft.au=Zhi%2C+M.%3B+Ding%2C+E.L.%3B+Theisen-Toupal%2C+J.+et+al.&rft.date=2013&rft.volume=8&rft.issue=11&rft.pages=e78962&rft_id=info:doi\/10.1371%2Fjournal.pone.0078962&rft_id=info:pmc\/PMC3829815&rft_id=info:pmid\/24260139&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3829815&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElwenspoekWhat20-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ElwenspoekWhat20_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Elwenspoek, M.M.C.; Scott, L.J.; Alsop, K. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7759753\" target=\"_blank\">\"What methods are being used to create an evidence base on the use of laboratory tests to monitor long-term conditions in primary care? A scoping review\"<\/a>. <i>Family Practice<\/i> <b>37<\/b> (6): 845-853. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Ffampra%2Fcmaa074\" target=\"_blank\">10.1093\/fampra\/cmaa074<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7759753\/\" target=\"_blank\">PMC7759753<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32820328\" target=\"_blank\">32820328<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7759753\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7759753<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+methods+are+being+used+to+create+an+evidence+base+on+the+use+of+laboratory+tests+to+monitor+long-term+conditions+in+primary+care%3F+A+scoping+review&rft.jtitle=Family+Practice&rft.aulast=Elwenspoek%2C+M.M.C.%3B+Scott%2C+L.J.%3B+Alsop%2C+K.+et+al.&rft.au=Elwenspoek%2C+M.M.C.%3B+Scott%2C+L.J.%3B+Alsop%2C+K.+et+al.&rft.date=2020&rft.volume=37&rft.issue=6&rft.pages=845-853&rft_id=info:doi\/10.1093%2Ffampra%2Fcmaa074&rft_id=info:pmc\/PMC7759753&rft_id=info:pmid\/32820328&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7759753&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScowenEval20-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScowenEval20_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scowen, C.; Wabe, N.; Eigenstetter, A. et al. (2020). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7552857\" target=\"_blank\">\"Evaluating the long-term effects of a data-driven approach to reduce variation in emergency department pathology investigations: study protocol for evaluation of the NSW Health Pathology Atlas of variation\"<\/a>. <i>BMJ Open<\/i> <b>10<\/b> (10): e039437. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmjopen-2020-039437\" target=\"_blank\">10.1136\/bmjopen-2020-039437<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7552857\/\" target=\"_blank\">PMC7552857<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/33046472\" target=\"_blank\">33046472<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7552857\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7552857<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluating+the+long-term+effects+of+a+data-driven+approach+to+reduce+variation+in+emergency+department+pathology+investigations%3A+study+protocol+for+evaluation+of+the+NSW+Health+Pathology+Atlas+of+variation&rft.jtitle=BMJ+Open&rft.aulast=Scowen%2C+C.%3B+Wabe%2C+N.%3B+Eigenstetter%2C+A.+et+al.&rft.au=Scowen%2C+C.%3B+Wabe%2C+N.%3B+Eigenstetter%2C+A.+et+al.&rft.date=2020&rft.volume=10&rft.issue=10&rft.pages=e039437&rft_id=info:doi\/10.1136%2Fbmjopen-2020-039437&rft_id=info:pmc\/PMC7552857&rft_id=info:pmid\/33046472&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7552857&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CampbellUnder20-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CampbellUnder20_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-CampbellUnder20_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Campbell, C.A.; Ling, L.; Kotwal, S. et al. (2020). \"Under-detection of acute kidney injury in hospitalised patients: A retrospective, multi-site, longitudinal study\". <i>Internal Medicine Journal<\/i> <b>50<\/b> (3): 307-314. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fimj.14264\" target=\"_blank\">10.1111\/imj.14264<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30816607\" target=\"_blank\">30816607<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Under-detection+of+acute+kidney+injury+in+hospitalised+patients%3A+A+retrospective%2C+multi-site%2C+longitudinal+study&rft.jtitle=Internal+Medicine+Journal&rft.aulast=Campbell%2C+C.A.%3B+Ling%2C+L.%3B+Kotwal%2C+S.+et+al.&rft.au=Campbell%2C+C.A.%3B+Ling%2C+L.%3B+Kotwal%2C+S.+et+al.&rft.date=2020&rft.volume=50&rft.issue=3&rft.pages=307-314&rft_id=info:doi\/10.1111%2Fimj.14264&rft_id=info:pmid\/30816607&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeorgiouMonit13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeorgiouMonit13_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Georgiou, A.; Vecellio, E.; Toouli, G. et al. (2013). \"Monitoring the impact of the electronic medical record on the quality of laboratory test ordering practices\". <i>Studies in Health Technologies and Informatics<\/i> <b>188<\/b>: 33\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23823285\" target=\"_blank\">23823285<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Monitoring+the+impact+of+the+electronic+medical+record+on+the+quality+of+laboratory+test+ordering+practices&rft.jtitle=Studies+in+Health+Technologies+and+Informatics&rft.aulast=Georgiou%2C+A.%3B+Vecellio%2C+E.%3B+Toouli%2C+G.+et+al.&rft.au=Georgiou%2C+A.%3B+Vecellio%2C+E.%3B+Toouli%2C+G.+et+al.&rft.date=2013&rft.volume=188&rft.pages=33%E2%80%938&rft_id=info:pmid\/23823285&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VecellioExam15-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VecellioExam15_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Vecellio, E.; Li, L.; Xiong, J. et al. (2015) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www1.health.gov.au\/internet\/main\/publishing.nsf\/Content\/0E3978777CE9637BCA2580C200088F78\/$File\/Final%20Report%20-%20Variation%20in%20Hospital%20Pathology%20-%20Macquarie%20University.pdf\" target=\"_blank\"><i>Examination of variation in hospital pathology investigations by Diagnosis-Related Groups and associations with outcomes and costs<\/i><\/a>. Centre for Health Systems and Safety Research. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781741384307<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www1.health.gov.au\/internet\/main\/publishing.nsf\/Content\/0E3978777CE9637BCA2580C200088F78\/$File\/Final%20Report%20-%20Variation%20in%20Hospital%20Pathology%20-%20Macquarie%20University.pdf\" target=\"_blank\">https:\/\/www1.health.gov.au\/internet\/main\/publishing.nsf\/Content\/0E3978777CE9637BCA2580C200088F78\/$File\/Final%20Report%20-%20Variation%20in%20Hospital%20Pathology%20-%20Macquarie%20University.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Examination+of+variation+in+hospital+pathology+investigations+by+Diagnosis-Related+Groups+and+associations+with+outcomes+and+costs&rft.aulast=Vecellio%2C+E.%3B+Li%2C+L.%3B+Xiong%2C+J.+et+al.&rft.au=Vecellio%2C+E.%3B+Li%2C+L.%3B+Xiong%2C+J.+et+al.&rft.date=2015&rft.pub=Centre+for+Health+Systems+and+Safety+Research&rft.isbn=9781741384307&rft_id=https%3A%2F%2Fwww1.health.gov.au%2Finternet%2Fmain%2Fpublishing.nsf%2FContent%2F0E3978777CE9637BCA2580C200088F78%2F%24File%2FFinal%2520Report%2520-%2520Variation%2520in%2520Hospital%2520Pathology%2520-%2520Macquarie%2520University.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WabeTheImp19-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WabeTheImp19_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wabe, N.; Li, L.; Lindeman, R. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6617970\" target=\"_blank\">\"The impact of rapid molecular diagnostic testing for respiratory viruses on outcomes for emergency department patients\"<\/a>. <i>Medical Journal of Australia<\/i> <b>210<\/b> (7): 316\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5694%2Fmja2.50049\" target=\"_blank\">10.5694\/mja2.50049<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6617970\/\" target=\"_blank\">PMC6617970<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30838671\" target=\"_blank\">30838671<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6617970\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6617970<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+impact+of+rapid+molecular+diagnostic+testing+for+respiratory+viruses+on+outcomes+for+emergency+department+patients&rft.jtitle=Medical+Journal+of+Australia&rft.aulast=Wabe%2C+N.%3B+Li%2C+L.%3B+Lindeman%2C+R.+et+al.&rft.au=Wabe%2C+N.%3B+Li%2C+L.%3B+Lindeman%2C+R.+et+al.&rft.date=2019&rft.volume=210&rft.issue=7&rft.pages=316%E2%80%9320&rft_id=info:doi\/10.5694%2Fmja2.50049&rft_id=info:pmc\/PMC6617970&rft_id=info:pmid\/30838671&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6617970&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JhaSumm08-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JhaSumm08_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">World Alliance for Patient Safety (2008). Jha, A.. ed. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/apps.who.int\/iris\/handle\/10665\/43874\" target=\"_blank\"><i>Summary of the evidence on patient safety: Implications for research<\/i><\/a>. World Health Organization. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9789241596541<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/apps.who.int\/iris\/handle\/10665\/43874\" target=\"_blank\">https:\/\/apps.who.int\/iris\/handle\/10665\/43874<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Summary+of+the+evidence+on+patient+safety%3A+Implications+for+research&rft.aulast=World+Alliance+for+Patient+Safety&rft.au=World+Alliance+for+Patient+Safety&rft.date=2008&rft.pub=World+Health+Organization&rft.isbn=9789241596541&rft_id=https%3A%2F%2Fapps.who.int%2Firis%2Fhandle%2F10665%2F43874&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CallenTheSafe11-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CallenTheSafe11_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Callen, J.; Georgiou, A.; Li, J. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3038104\" target=\"_blank\">\"The safety implications of missed test results for hospitalised patients: A systematic review\"<\/a>. <i>BMJ Quality and Safety<\/i> <b>20<\/b> (2): 194-9. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmjqs.2010.044339\" target=\"_blank\">10.1136\/bmjqs.2010.044339<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3038104\/\" target=\"_blank\">PMC3038104<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21300992\" target=\"_blank\">21300992<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3038104\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3038104<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+safety+implications+of+missed+test+results+for+hospitalised+patients%3A+A+systematic+review&rft.jtitle=BMJ+Quality+and+Safety&rft.aulast=Callen%2C+J.%3B+Georgiou%2C+A.%3B+Li%2C+J.+et+al.&rft.au=Callen%2C+J.%3B+Georgiou%2C+A.%3B+Li%2C+J.+et+al.&rft.date=2011&rft.volume=20&rft.issue=2&rft.pages=194-9&rft_id=info:doi\/10.1136%2Fbmjqs.2010.044339&rft_id=info:pmc\/PMC3038104&rft_id=info:pmid\/21300992&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3038104&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NQFImprov17-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NQFImprov17_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-NQFImprov17_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Quality Forum (September 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.qualityforum.org\/Publications\/2017\/09\/Improving_Diagnostic_Quality_and_Safety_Final_Report.aspx\" target=\"_blank\">\"Improving Diagnostic Quality and Safety Final Report\"<\/a>. National Quality Forum<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.qualityforum.org\/Publications\/2017\/09\/Improving_Diagnostic_Quality_and_Safety_Final_Report.aspx\" target=\"_blank\">https:\/\/www.qualityforum.org\/Publications\/2017\/09\/Improving_Diagnostic_Quality_and_Safety_Final_Report.aspx<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 May 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Improving+Diagnostic+Quality+and+Safety+Final+Report&rft.atitle=&rft.aulast=National+Quality+Forum&rft.au=National+Quality+Forum&rft.date=September+2017&rft.pub=National+Quality+Forum&rft_id=https%3A%2F%2Fwww.qualityforum.org%2FPublications%2F2017%2F09%2FImproving_Diagnostic_Quality_and_Safety_Final_Report.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KripalaniPromo07-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KripalaniPromo07_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kripalani, S.; Jackson, A.T.; Schnipper, J.L et al. (2007). \"Promoting effective transitions of care at hospital discharge: A review of key issues for hospitalists\". <i>Journal of Hospital Medicine<\/i> <b>2<\/b> (5): 314-23. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fjhm.228\" target=\"_blank\">10.1002\/jhm.228<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17935242\" target=\"_blank\">17935242<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Promoting+effective+transitions+of+care+at+hospital+discharge%3A+A+review+of+key+issues+for+hospitalists&rft.jtitle=Journal+of+Hospital+Medicine&rft.aulast=Kripalani%2C+S.%3B+Jackson%2C+A.T.%3B+Schnipper%2C+J.L+et+al.&rft.au=Kripalani%2C+S.%3B+Jackson%2C+A.T.%3B+Schnipper%2C+J.L+et+al.&rft.date=2007&rft.volume=2&rft.issue=5&rft.pages=314-23&rft_id=info:doi\/10.1002%2Fjhm.228&rft_id=info:pmid\/17935242&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MooreMed03-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MooreMed03_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moore, C.; Wisnivesky, J.; Williams, S. et al. (2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1494907\" target=\"_blank\">\"Medical errors related to discontinuity of care from an inpatient to an outpatient setting\"<\/a>. <i>Journal of General Internal Medicine<\/i> <b>18<\/b> (8): 646-51. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1046%2Fj.1525-1497.2003.20722.x\" target=\"_blank\">10.1046\/j.1525-1497.2003.20722.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1494907\/\" target=\"_blank\">PMC1494907<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12911647\" target=\"_blank\">12911647<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1494907\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1494907<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+errors+related+to+discontinuity+of+care+from+an+inpatient+to+an+outpatient+setting&rft.jtitle=Journal+of+General+Internal+Medicine&rft.aulast=Moore%2C+C.%3B+Wisnivesky%2C+J.%3B+Williams%2C+S.+et+al.&rft.au=Moore%2C+C.%3B+Wisnivesky%2C+J.%3B+Williams%2C+S.+et+al.&rft.date=2003&rft.volume=18&rft.issue=8&rft.pages=646-51&rft_id=info:doi\/10.1046%2Fj.1525-1497.2003.20722.x&rft_id=info:pmc\/PMC1494907&rft_id=info:pmid\/12911647&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1494907&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarayonIncorp10-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CarayonIncorp10_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Carayon, P.; Karsh, B.T.; Cartmill, R. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/digital.ahrq.gov\/ahrq-funded-projects\/incorporating-health-information-technology-workflow-redesign\/citation\/workflow\" target=\"_blank\">\"Incorporating health information technology into workflow redesign: Request for information summary report\"<\/a>. Agency for Healthcare Research and Quality<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/digital.ahrq.gov\/ahrq-funded-projects\/incorporating-health-information-technology-workflow-redesign\/citation\/workflow\" target=\"_blank\">https:\/\/digital.ahrq.gov\/ahrq-funded-projects\/incorporating-health-information-technology-workflow-redesign\/citation\/workflow<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Incorporating+health+information+technology+into+workflow+redesign%3A+Request+for+information+summary+report&rft.atitle=&rft.aulast=Carayon%2C+P.%3B+Karsh%2C+B.T.%3B+Cartmill%2C+R.+et+al.&rft.au=Carayon%2C+P.%3B+Karsh%2C+B.T.%3B+Cartmill%2C+R.+et+al.&rft.date=2010&rft.pub=Agency+for+Healthcare+Research+and+Quality&rft_id=https%3A%2F%2Fdigital.ahrq.gov%2Fahrq-funded-projects%2Fincorporating-health-information-technology-workflow-redesign%2Fcitation%2Fworkflow&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CampbellTowards12-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CampbellTowards12_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Campbell, C.; Horvath, A. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3529552\" target=\"_blank\">\"Towards harmonisation of critical laboratory result management - Review of the literature and survey of australasian practices\"<\/a>. <i>Clinical Biochemist Reviews<\/i> <b>33<\/b> (4): 149\u201360. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3529552\/\" target=\"_blank\">PMC3529552<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23267247\" target=\"_blank\">23267247<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3529552\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3529552<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+harmonisation+of+critical+laboratory+result+management+-+Review+of+the+literature+and+survey+of+australasian+practices&rft.jtitle=Clinical+Biochemist+Reviews&rft.aulast=Campbell%2C+C.%3B+Horvath%2C+A.&rft.au=Campbell%2C+C.%3B+Horvath%2C+A.&rft.date=2012&rft.volume=33&rft.issue=4&rft.pages=149%E2%80%9360&rft_id=info:pmc\/PMC3529552&rft_id=info:pmid\/23267247&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3529552&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CampbellWhat16-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CampbellWhat16_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Campbell, C.; Georgiou, A.; Westbrook, J.I. et al. (2016). \"What Alert Thresholds Should Be Used to Identify Critical Risk Results: A Systematic Review of the Evidence\". <i>Clinical Chemistry<\/i> <b>62<\/b> (11): 1445\u201357. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1373%2Fclinchem.2016.260638\" target=\"_blank\">10.1373\/clinchem.2016.260638<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27565161\" target=\"_blank\">27565161<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+Alert+Thresholds+Should+Be+Used+to+Identify+Critical+Risk+Results%3A+A+Systematic+Review+of+the+Evidence&rft.jtitle=Clinical+Chemistry&rft.aulast=Campbell%2C+C.%3B+Georgiou%2C+A.%3B+Westbrook%2C+J.I.+et+al.&rft.au=Campbell%2C+C.%3B+Georgiou%2C+A.%3B+Westbrook%2C+J.I.+et+al.&rft.date=2016&rft.volume=62&rft.issue=11&rft.pages=1445%E2%80%9357&rft_id=info:doi\/10.1373%2Fclinchem.2016.260638&rft_id=info:pmid\/27565161&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PITUSJuly13-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PITUSJuly13_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pathology Information, Terminology and Units Standardisation Program (July 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.rcpa.edu.au\/Library\/Practising-Pathology\/PTIS\/PITUS\/Docs\/PITUS-Update-Issue-1-July-2013\" target=\"_blank\">\"PITUS Update - Newsletter of the RCPA Pathology Information, Terminology and Units Standardisation Program\"<\/a> (PDF). <i>PITUS Update<\/i> (Royal College of Pathologists of Australasia) (1)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.rcpa.edu.au\/Library\/Practising-Pathology\/PTIS\/PITUS\/Docs\/PITUS-Update-Issue-1-July-2013\" target=\"_blank\">https:\/\/www.rcpa.edu.au\/Library\/Practising-Pathology\/PTIS\/PITUS\/Docs\/PITUS-Update-Issue-1-July-2013<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PITUS+Update+-+Newsletter+of+the+RCPA+Pathology+Information%2C+Terminology+and+Units+Standardisation+Program&rft.jtitle=PITUS+Update&rft.aulast=Pathology+Information%2C+Terminology+and+Units+Standardisation+Program&rft.au=Pathology+Information%2C+Terminology+and+Units+Standardisation+Program&rft.date=July+2013&rft.issue=1&rft.pub=Royal+College+of+Pathologists+of+Australasia&rft_id=https%3A%2F%2Fwww.rcpa.edu.au%2FLibrary%2FPractising-Pathology%2FPTIS%2FPITUS%2FDocs%2FPITUS-Update-Issue-1-July-2013&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HardieInformatics18-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HardieInformatics18_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hardie, R.-A.; Moore, D.; Holzhauser, D. et al. (2018). \"Informatics External Quality Assurance (IEQA) Down Under: evaluation of a pilot implementation\". <i>Journal of Laboratory Medicine<\/i> <b>42<\/b> (6): 297-304. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Flabmed-2018-0050\" target=\"_blank\">10.1515\/labmed-2018-0050<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Informatics+External+Quality+Assurance+%28IEQA%29+Down+Under%3A+evaluation+of+a+pilot+implementation&rft.jtitle=Journal+of+Laboratory+Medicine&rft.aulast=Hardie%2C+R.-A.%3B+Moore%2C+D.%3B+Holzhauser%2C+D.+et+al.&rft.au=Hardie%2C+R.-A.%3B+Moore%2C+D.%3B+Holzhauser%2C+D.+et+al.&rft.date=2018&rft.volume=42&rft.issue=6&rft.pages=297-304&rft_id=info:doi\/10.1515%2Flabmed-2018-0050&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JoostenSys08-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JoostenSys08_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Joosten, E.A.G.; DeFuentes-Merillas, L.; de Weert, G.H. et al. (2008). \"Systematic review of the effects of shared decision-making on patient satisfaction, treatment adherence and health status\". <i>Psychotherapy and Psychosomatics<\/i> <b>77<\/b> (4): 219\u201326. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1159%2F000126073\" target=\"_blank\">10.1159\/000126073<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18418028\" target=\"_blank\">18418028<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Systematic+review+of+the+effects+of+shared+decision-making+on+patient+satisfaction%2C+treatment+adherence+and+health+status&rft.jtitle=Psychotherapy+and+Psychosomatics&rft.aulast=Joosten%2C+E.A.G.%3B+DeFuentes-Merillas%2C+L.%3B+de+Weert%2C+G.H.+et+al.&rft.au=Joosten%2C+E.A.G.%3B+DeFuentes-Merillas%2C+L.%3B+de+Weert%2C+G.H.+et+al.&rft.date=2008&rft.volume=77&rft.issue=4&rft.pages=219%E2%80%9326&rft_id=info:doi\/10.1159%2F000126073&rft_id=info:pmid\/18418028&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-deSilvaHelp14-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-deSilvaHelp14_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">de Silva, D. (March 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.health.org.uk\/publications\/helping-measure-person-centred-care\" target=\"_blank\">\"Helping measure person-centred care: A review of evidence about commonly used approaches and tools used to help measure person-centred care\"<\/a>. <i>The Health Foundation<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.health.org.uk\/publications\/helping-measure-person-centred-care\" target=\"_blank\">https:\/\/www.health.org.uk\/publications\/helping-measure-person-centred-care<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Helping+measure+person-centred+care%3A+A+review+of+evidence+about+commonly+used+approaches+and+tools+used+to+help+measure+person-centred+care&rft.atitle=The+Health+Foundation&rft.aulast=de+Silva%2C+D.&rft.au=de+Silva%2C+D.&rft.date=March+2014&rft_id=https%3A%2F%2Fwww.health.org.uk%2Fpublications%2Fhelping-measure-person-centred-care&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OsheroffImprov05-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OsheroffImprov05_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Osheroff, J.; Pifer, E.; Teich, J. et al. (2005). <i>Improving Outcomes with Clinical Decision Support: An Implementer's Guide<\/i> (1st ed.). Productivity Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780976127727.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Improving+Outcomes+with+Clinical+Decision+Support%3A+An+Implementer%27s+Guide&rft.aulast=Osheroff%2C+J.%3B+Pifer%2C+E.%3B+Teich%2C+J.+et+al.&rft.au=Osheroff%2C+J.%3B+Pifer%2C+E.%3B+Teich%2C+J.+et+al.&rft.date=2005&rft.edition=1st&rft.pub=Productivity+Press&rft.isbn=9780976127727&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RCPAPath13-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RCPAPath13_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Australasian College for Emergency Medicine, Royal College of Pathologists of Australasia (December 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.rcpa.edu.au\/Library\/College-Policies\/Guidelines\/Pathology-Testing-in-the-Emergency-Department.aspx\" target=\"_blank\">\"Pathology Testing in the Emergency Department\"<\/a> (PDF). Royal College of Pathologists of Australasia<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.rcpa.edu.au\/Library\/College-Policies\/Guidelines\/Pathology-Testing-in-the-Emergency-Department.aspx\" target=\"_blank\">https:\/\/www.rcpa.edu.au\/Library\/College-Policies\/Guidelines\/Pathology-Testing-in-the-Emergency-Department.aspx<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Pathology+Testing+in+the+Emergency+Department&rft.atitle=&rft.aulast=Australasian+College+for+Emergency+Medicine%2C+Royal+College+of+Pathologists+of+Australasia&rft.au=Australasian+College+for+Emergency+Medicine%2C+Royal+College+of+Pathologists+of+Australasia&rft.date=December+2018&rft.pub=Royal+College+of+Pathologists+of+Australasia&rft_id=https%3A%2F%2Fwww.rcpa.edu.au%2FLibrary%2FCollege-Policies%2FGuidelines%2FPathology-Testing-in-the-Emergency-Department.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GargEffects05-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GargEffects05_29-0\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-GargEffects05_29-1\">29.1<\/a><\/sup> <sup><a href=\"#cite_ref-GargEffects05_29-2\">29.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Garg, A.X.; Adhikari, N.K.J.; McDonald, H. et al. (2005). \"Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: A systematic review\". <i>JAMA<\/i> <b>293<\/b> (10): 1223-38. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.293.10.1223\" target=\"_blank\">10.1001\/jama.293.10.1223<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15755945\" target=\"_blank\">15755945<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Effects+of+computerized+clinical+decision+support+systems+on+practitioner+performance+and+patient+outcomes%3A+A+systematic+review&rft.jtitle=JAMA&rft.aulast=Garg%2C+A.X.%3B+Adhikari%2C+N.K.J.%3B+McDonald%2C+H.+et+al.&rft.au=Garg%2C+A.X.%3B+Adhikari%2C+N.K.J.%3B+McDonald%2C+H.+et+al.&rft.date=2005&rft.volume=293&rft.issue=10&rft.pages=1223-38&rft_id=info:doi\/10.1001%2Fjama.293.10.1223&rft_id=info:pmid\/15755945&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeorgiouEhealth19-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeorgiouEhealth19_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Georgiou, A.; Prgomet, M. (2019). \"E-health: Developing trust, confidence, quality and sustainability\". <i>Health Information Management<\/i> <b>48<\/b> (2): 59\u201361. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1833358318808064\" target=\"_blank\">10.1177\/1833358318808064<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30392402\" target=\"_blank\">30392402<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=E-health%3A+Developing+trust%2C+confidence%2C+quality+and+sustainability&rft.jtitle=Health+Information+Management&rft.aulast=Georgiou%2C+A.%3B+Prgomet%2C+M.&rft.au=Georgiou%2C+A.%3B+Prgomet%2C+M.&rft.date=2019&rft.volume=48&rft.issue=2&rft.pages=59%E2%80%9361&rft_id=info:doi\/10.1177%2F1833358318808064&rft_id=info:pmid\/30392402&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MiaoManag20-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MiaoManag20_31-0\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-MiaoManag20_31-1\">31.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Miao, M.; Dahm, M.R.; Li, J. et al. (2020). \"Managing Uncertainty During the Communication of Diagnostic Test Information Between Patients and Clinicians in Australian Emergency Care\". <i>Qualitative Health Research<\/i> <b>30<\/b> (8): 1287-1300. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1049732320913037\" target=\"_blank\">10.1177\/1049732320913037<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32249721\" target=\"_blank\">32249721<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Managing+Uncertainty+During+the+Communication+of+Diagnostic+Test+Information+Between+Patients+and+Clinicians+in+Australian+Emergency+Care&rft.jtitle=Qualitative+Health+Research&rft.aulast=Miao%2C+M.%3B+Dahm%2C+M.R.%3B+Li%2C+J.+et+al.&rft.au=Miao%2C+M.%3B+Dahm%2C+M.R.%3B+Li%2C+J.+et+al.&rft.date=2020&rft.volume=30&rft.issue=8&rft.pages=1287-1300&rft_id=info:doi\/10.1177%2F1049732320913037&rft_id=info:pmid\/32249721&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HirschbergAdvan15-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HirschbergAdvan15_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hirschberg, J.; Manning, C.D. (2015). \"Advances in natural language processing\". <i>Science<\/i> <b>349<\/b> (6245): 261\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.aaa8685\" target=\"_blank\">10.1126\/science.aaa8685<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26185244\" target=\"_blank\">26185244<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Advances+in+natural+language+processing&rft.jtitle=Science&rft.aulast=Hirschberg%2C+J.%3B+Manning%2C+C.D.&rft.au=Hirschberg%2C+J.%3B+Manning%2C+C.D.&rft.date=2015&rft.volume=349&rft.issue=6245&rft.pages=261%E2%80%936&rft_id=info:doi\/10.1126%2Fscience.aaa8685&rft_id=info:pmid\/26185244&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeorgiouTheImp19-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GeorgiouTheImp19_33-0\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-GeorgiouTheImp19_33-1\">33.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Georgiou, A.; Li, J.; Thomas, J. et al. (2019). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6562156\" target=\"_blank\">\"The impact of health information technology on the management and follow-up of test results - A systematic review\"<\/a>. <i>JAMIA<\/i> <b>26<\/b> (7): 678\u2013688. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fjamia%2Focz032\" target=\"_blank\">10.1093\/jamia\/ocz032<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6562156\/\" target=\"_blank\">PMC6562156<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31192362\" target=\"_blank\">31192362<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6562156\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6562156<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+impact+of+health+information+technology+on+the+management+and+follow-up+of+test+results+-+A+systematic+review&rft.jtitle=JAMIA&rft.aulast=Georgiou%2C+A.%3B+Li%2C+J.%3B+Thomas%2C+J.+et+al.&rft.au=Georgiou%2C+A.%3B+Li%2C+J.%3B+Thomas%2C+J.+et+al.&rft.date=2019&rft.volume=26&rft.issue=7&rft.pages=678%E2%80%93688&rft_id=info:doi\/10.1093%2Fjamia%2Focz032&rft_id=info:pmc\/PMC6562156&rft_id=info:pmid\/31192362&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6562156&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeorgiouTime11-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeorgiouTime11_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Georgiou, A.; Westbrook, J.I.; Braithwaite, J. (2011). \"Time matters: A theoretical and empirical examination of the temporal landscape of a hospital pathology service and the impact of e-health\". <i>Social Science and Medicine<\/i> <b>72<\/b> (10): 1603-10. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.socscimed.2011.03.020\" target=\"_blank\">10.1016\/j.socscimed.2011.03.020<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21497430\" target=\"_blank\">21497430<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Time+matters%3A+A+theoretical+and+empirical+examination+of+the+temporal+landscape+of+a+hospital+pathology+service+and+the+impact+of+e-health&rft.jtitle=Social+Science+and+Medicine&rft.aulast=Georgiou%2C+A.%3B+Westbrook%2C+J.I.%3B+Braithwaite%2C+J.&rft.au=Georgiou%2C+A.%3B+Westbrook%2C+J.I.%3B+Braithwaite%2C+J.&rft.date=2011&rft.volume=72&rft.issue=10&rft.pages=1603-10&rft_id=info:doi\/10.1016%2Fj.socscimed.2011.03.020&rft_id=info:pmid\/21497430&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeorgiouAnEmp12-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeorgiouAnEmp12_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Georgiou, A.; Westbrook, J.I.; Braithwaite, J. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3407796\" target=\"_blank\">\"An empirically-derived approach for investigating Health Information Technology: The Elementally Entangled Organisational Communication (EEOC) framework\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>12<\/b>: 68. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-12-68\" target=\"_blank\">10.1186\/1472-6947-12-68<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3407796\/\" target=\"_blank\">PMC3407796<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22788698\" target=\"_blank\">22788698<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3407796\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3407796<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+empirically-derived+approach+for+investigating+Health+Information+Technology%3A+The+Elementally+Entangled+Organisational+Communication+%28EEOC%29+framework&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Georgiou%2C+A.%3B+Westbrook%2C+J.I.%3B+Braithwaite%2C+J.&rft.au=Georgiou%2C+A.%3B+Westbrook%2C+J.I.%3B+Braithwaite%2C+J.&rft.date=2012&rft.volume=12&rft.pages=68&rft_id=info:doi\/10.1186%2F1472-6947-12-68&rft_id=info:pmc\/PMC3407796&rft_id=info:pmid\/22788698&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3407796&rfr_id=info:sid\/en.wikipedia.org:Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150554\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.733 seconds\nReal time usage: 1.353 seconds\nPreprocessor visited node count: 27796\/1000000\nPreprocessor generated node count: 38217\/1000000\nPost\u2010expand include size: 246815\/2097152 bytes\nTemplate argument size: 84310\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 581.802 1 - -total\n 87.37% 508.307 1 - Template:Reflist\n 73.66% 428.580 35 - Template:Citation\/core\n 54.65% 317.964 23 - Template:Cite_journal\n 12.33% 71.715 7 - Template:Cite_web\n 10.47% 60.912 5 - Template:Cite_book\n 8.97% 52.173 58 - Template:Citation\/identifier\n 6.82% 39.685 1 - Template:Infobox_journal_article\n 6.44% 37.467 1 - Template:Infobox\n 4.58% 26.647 38 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:12610-0!*!0!!en!5!* and timestamp 20210705150552 and revision id 43249\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value\">https:\/\/www.limswiki.org\/index.php\/Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n\n<\/body>","236fa37968f1db59b5299967fce1aa76_images":["https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/1\/1f\/Fig1_Georgiou_FrontDigHlth2021_3.jpg","https:\/\/s3.limswiki.org\/www.limswiki.org\/images\/9\/94\/Tab1_Georgiou_FrontDigHlth2021_3.jpg"],"236fa37968f1db59b5299967fce1aa76_timestamp":1625497552,"a3b638adc3dd9824430235085d4b072b_type":"article","a3b638adc3dd9824430235085d4b072b_title":"Accelerated solvent extraction of terpenes in cannabis coupled with various injection techniques for GC-MS analysis (Myers et al. 2021)","a3b638adc3dd9824430235085d4b072b_url":"https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis","a3b638adc3dd9824430235085d4b072b_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Accelerated solvent extraction of terpenes in cannabis coupled with various injection techniques for GC-MS analysis\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom CannaQAWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nAccelerated solvent extraction of terpenes in cannabis coupled with various injection techniques for GC-MS analysisJournal\n \nFrontiers in ChemistryAuthor(s)\n \nMyers, Colton; Herrington, Jason S.; Hamrah, Paul; Anderson, KelseyAuthor affiliation(s)\n \nRestek Corporation, Verity AnalyticsPrimary contact\n \ncolton dot myers at restek dot comEditors\n \nGionfriddo, EmanuelaYear published\n \n2021Volume and issue\n \n9Article #\n \n619770DOI\n \n10.3389\/fchem.2021.619770ISSN\n \n2296-2646Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fchem.2021.619770\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fchem.2021.619770\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Experimental \n\n3.1 Materials and reagents \n3.2 HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow \n3.3 Terpene extraction evaluation \n3.4 DI-SPME Arrow vs. LI syringe \n\n3.4.1 Hops pellets and cannabis flower preparation \n3.4.2 Accelerated solvent extractor \n3.4.3 After ASE processing \n3.4.4 Terpenes standards and internal standards \n3.4.5 DI-SPME Arrow calibration \n3.4.6 LI syringe calibration \n3.4.7 Method validation and chemovar experiments \n\n\n\n\n4 Results and discussion \n\n4.1 HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow \n4.2 Hand shakeout vs. accelerated solvent extractor \n4.3 Hops for clean surrogate matrix \n4.4 Method performance (DI-SPME Arrow vs. LI syringe) \n4.5 DI-SPME Arrow terpene validation \n4.6 Liquid injection syringe terpene validation \n4.7 Terpene chemovar evaluation with DI-SPME Arrow and LI syringe \n\n\n5 Conclusions, future work, and recommendations \n6 Supplementary material \n7 Acknowledgements \n\n7.1 Author contributions \n7.2 Data availability statement \n7.3 Conflict of interest \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nThe cannabis market is expanding exponentially in the United States. As state-wide legalization efforts increase, so also do demands for analytical testing methodologies. One of the main tests conducted on cannabis products is the analysis for terpenes. This research focused on implementation of accelerated solvent extraction (ASE), utilizing surrogate matrix matching, and evaluation of traditional vs. more modern sample introduction techniques for analyzing terpenes via gas chromatography\u2013mass spectrometry (GC-MS). Introduction techniques included headspace syringe (HS syringe), HS-solid-phase microextraction Arrow (HS-SPME Arrow), direct immersion-SPME Arrow (DI-SPME Arrow), and liquid injection syringe (LI syringe). The LI syringe approach was deemed the most straightforward and robust method, with terpene working ranges of 0.04\u20135.12 \u03bcg\/mL; r2 values of 0.988\u20130.996 (0.993 average); limit of quantitation values of 0.017\u20130.129 \u03bcg\/mL (0.047 average); analytical precisions of 2.58\u20139.64% RSD (1.56 average); overall ASE-LI-syringe-GC-MS method precisions of 1.73\u201314.6% RSD (4.97 average); and % recoveries of 84.6\u201398.9% (90.2 average) for the 23 terpenes of interest. Sample workflows and results are discussed, with an evaluation of the advantages\/limitations of each approach and opportunities for future work.\nKeywords: accelerated solvent extraction (ASE), terpenes, solid-phase microextraction (SPME), solid-phase microextraction Arrow (SPME Arrow), gas chromatography\u2013mass spectrometry (GC-MS)\n\nIntroduction \nThe legal cannabis market is one of the fastest growing markets across the globe. In 2019, cannabis use for medicinal purposes in the United States generated $4 billion to $4.9 billion in sales, compared to the adult-use estimates between $6.6 billion and $8.1 billion.[1] As the United States and additional countries continue to legalize the use of medicinal and recreational cannabis, analytical testing demands increase. A 2020 report by Market Data Forecast valued the global cannabis testing market at $1,218.0 million in 2019 and estimated it to be growing at a compound annual growth rate (CAGR) of 12.42%.[2] The market is projected to almost double at $2,187.3 million by 2024.[2] Of the examinations conducted in cannabis testing laboratories, terpene profiling is a popular analysis, regardless of state regulations. \nTerpenes are a naturally occurring set of organic compounds, which are commonly found in plants, and are typically strong in odor.[3] Terpenes are made up of isoprene units and are classified by the number of their isoprene units.[3] The two types of terpenes that are commonly analyzed in the cannabis testing industry are monoterpenes, which have two isoprene units, and sesquiterpenes, which have three isoprene units. Over 100 terpenes have been identified in different cannabis chemical varieties (chemovars).[4] Each cannabis chemovar has its own unique terpene profile, giving consumers different aromas, flavors, and experiences depending on the chemovar they use. According to Russo et al., terpenes play a major role in the entourage effect, which is the synergistic interaction between phytocannabinoids and terpenoids with respect to treating numerous ailments (e.g., depression).[5] The desire to understand and capitalize on this entourage effect is the motivation for terpene testing in the cannabis industry.\nTerpenes have been analyzed in numerous commodities within the food and beverage industry. Previous studies have looked at a variety of matrices (e.g., tequila) and have used different analytical techniques (e.g., solid-phase microextraction [SPME]) to conduct the analysis.[4][6][7][8][9][10][11][12][13][14][15][16][17][18][19] However, only a few studies have shown the analysis of terpenes in cannabis and hemp matrices (e.g., flower, gummy), and their robustness for compliance laboratories remains uncertain. Calvi et al., Ternelli et al., Gaggotti et al., and Stenerson et al. did not perform extractions on cannabis and hemp samples; rather, they added the samples directly to a headspace (HS) vial and demonstrated the analysis of terpenes using HS-SPME.[4][10][15][18] Nguyen et al. utilized a pseudo extraction by adding a solvent to dried material, followed by analysis via headspace gas chromatography\u2013mass spectrometry (HS-GC-MS).[17] The five aforementioned studies appear to lack an exhaustive cannabis or hemp extraction, and therefore this calls into question the real-world applicability of these methods. Furthermore, Calvi et al., Ternelli et al., Gaggotti et al., and Stenerson et al. only focused on profiling the terpenes in the cannabis or hemp matrices studied and therefore only presented qualitative and semi-quantitative data.[4][10][15][18]\nBakro et al., Brown et al., Ibrahim et al., and Shapira et al. extracted cannabis flower with ethanol, hexane, ethyl acetate, and methanol, respectively, and provided quantitative results.[11][12][13][14] However, Bakro et al. only looked at hemp and used a nonspecific gas chromatography with flame-ionization detection (GC-FID) approach, which is cumbersome when attempting to differentiate between coeluting terpenes of interest and matrix interferences.[14] Brown et al. did not provide method accuracies for all targeted terpenes and reported less than desirable linearities, which fell below an r2 value of 0.960 for each terpene of interest.[11]\nTo date, the most promising methods\u2014presented by Ibrahim et al. and Shapira et al.\u2014utilize exhaustive cannabis and hemp extraction approaches, followed by GC-MS and reported desirable quantitative results.[12][13] Ibrahim et al. and Shapira et al. used sample introduction techniques like liquid injection without filtration and static headspace GC-MS (SHS-GC-MS), respectively. More importantly, none of the aforementioned studies accounted for matrix effects, as they all used solvent-based calibrations and, due to the complexity and dirtiness of cannabis matrices, this could lead to inaccurate reporting.[20] In addition, these studies did not evaluate more modern sample extraction approaches [e.g., accelerated solvent extraction (ASE)] or sample introduction techniques (e.g., direct immersion-SPME Arrow [DI-SPME Arrow]).\nThe following study was conducted to evaluate more modern sample preparation and introduction techniques and demonstrate their potential value to cannabis compliance testing laboratories in need of guidance for qualitative and quantitative terpenes analysis. In addition, this study evaluated accelerated solvent extraction (ASE 350) of terpenes in cannabis samples, which is commonly used in other markets within the analytical testing industry.[21][22][23][24] Furthermore, to avoid potentially inaccurate reporting, matrix-matched standards were used for calibration. Finally, the more traditional headspace syringe (HS syringe) and liquid injection syringe (LI syringe) approaches were compared to the more modern HS-solid-phase microextraction Arrow (HS-SPME Arrow) and DI-SPME Arrow, which have recently demonstrated enhanced robustness and improved sensitivity over traditional SPME fibers.[25]\n\nExperimental \nThe following experimental sections describe the detailed procedures utilized during the three main parts of this manuscript: \n1. HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow for the determination of the preferred sample introduction approach with the use of terpenes in solution;\n2. Terpene extraction evaluation for the evaluation of an ideal terpene extraction method for cannabis flower; and\n3. Combining the information gathered from the prior two procedures for a final comparison with an existing validated LI syringe technique (i.e., validated with the California Bureau of Cannabis Control [BCC]), outlined in DI-SPME Arrow vs. LI syringe.\n\nMaterials and reagents \nHop pellets were obtained from the Beverage Factory (San Diego, CA, United States) and stored at \u221210 \u00b0C for one hour. Dried cannabis material was obtained from Cream of the Crop (San Diego, CA, United States) and stored at \u221210 \u00b0C for one hour. Cannabis Terpene Standards #1 and #2 (cat# 34095 and 34096) were purchased from Restek Corporation (Bellefonte, PA, United States). Napthalene-d8 was purchased from Restek Corporation (Bellefonte, PA, United States). Isopropanol (IPA) was purchased from Filtrous (LCMS Grade) and 1.1 mm, 120 \u00b5m DVB\/PDMS SPME Arrows were purchased from Restek Corporation (Bellefonte, PA, United States).\n\nHS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow \nThe following experiments were conducted to evaluate the differences between HS syringe, HS-SPME Arrow, and DI-SPME Arrow as sample introduction techniques. Each technique was evaluated using Cannabis Terpene Standard #1 and #2 (cat# 34095 and 34096) from Restek Corporation (Bellefonte, PA, United States). Samples were evaluated using the same GC-MS conditions shown in Supplementary Table S1. For the HS syringe and HS-SPME Arrow samples, a standard stock solution was made by diluting both standards into one solution for a final concentration of 5 \u03bcg\/mL in IPA. Samples were prepared by adding 1.5 g of NaCl to a 20 mL HS vial, followed by 1 mL of 5 \u03bcg\/mL stock solution and 4 mL of water for a final concentration of 1 \u03bcg\/mL (see sampling conditions in Supplementary Tables S2, S3). Previous work (results not shown) demonstrated that HS-SPME Arrow analyte responses were higher and more reproducible when using an incubation temperature of 40 \u00b0C or less, hence having lower incubation temperature compared to the HS syringe method. The DI-SPME Arrow samples were prepared by diluting both standards into one stock solution for a final concentration of 20 \u03bcg\/mL. To a 20 mL HS vial, 1 mL of the 20 \u03bcg\/mL stock solution was added, followed by 19 mL of water for a final concentration of 1 \u03bcg\/mL (same concentration for HS syringe and HS-SPME Arrow; see sampling conditions in Supplementary Table S3). Each technique was run in triplicate for the initial evaluation of sample introduction techniques.\n\nTerpene extraction evaluation \nAn evaluation of the terpene extraction processes was conducted to understand the advantages and limitations of certain techniques. Extractions using the Dionex Accelerated Solvent Extractor (ASE 350) were compared to a hand-solvent extraction for terpene analysis. Three chemical varieties (chemovars) were used to evaluate both extraction techniques. Cannabis flower was frozen at \u221210 \u00b0C for one hour, then homogenized on a sheet pan with a rolling pin. For ASE 350 extractions, 0.5 g of homogenized cannabis flower was weighed and added to a 10 mL stainless steel ASE 350 cell, and the remaining cell volume was lightly packed with diatomaceous earth. The cell was then extracted using the parameters in Table 1. The extract was then diluted to 12 mL in a graduated cylinder due to both convenience and the fact that this approach achieved the desired data quality objectives of this study (e.g., method precision RSDs <15%). However, future researchers may consider the use of volumetric flasks to achieve better precision. One mL of the cannabis flower extract was added to a 2.5 mL autosampler vial and then analyzed. For hand extractions, 0.5 g of homogenized cannabis flower was weighed and added to a 50 mL centrifuge tube, followed by 12 mL of IPA. Samples were vortexed for three minutes and sonicated at 40 \u00b0C for five minutes. Samples were then centrifuged in a Sorvall RT7 Plus centrifuge for three minutes. One mL of the supernatant was added to a 2.5 mL autosampler vial and then analyzed. ASE 350 and hand extractions were analyzed via GC-FID.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 1. Accelerated solvent extractor (ASE) parameters for extracting terpenes from hops pellets and cannabis flower\n\n\n\nDI-SPME Arrow vs. LI syringe \nResults from the experiments outlined in the prior two steps indicated DI-SPME Arrow was the preferred sample introduction approach, and ASE was the ideal terpene extraction technique for cannabis. This information was then utilized for a comparison to an existing validated LI syringe method. However, the experiments conducted in HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow were carried out in Pennsylvania, and the use of cannabis flower necessitated a fully licensed laboratory, which was located in California. The DI-SPME Arrow parameters, outlined in Table 2, had been further optimized for terpenes analysis in cannabis and used in the California laboratory; however, they are only slightly different from the initial DI-SPME Arrow parameters, as outlined in Supplementary Table S3. In addition, a gas chromatography\u2013tandem mass spectrometry (GC-MS\/MS) method was used in single quad MS mode in the California laboratory, since single quad MS is more common for this analysis (see parameters in Table 3). Furthermore, a selected ion monitoring (SIM) method (Supplementary Table S4) was utilized to help eliminate background noise and provide better sensitivity. LI syringe was only evaluated in California using the parameters listed in Table 3.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 2. Optimized and final DI-SPME Arrow parameters\n\n\n\n\n\n\n\n\n\n\n\n\n Table 3. Analytical parameters for evaluating terpenes in cannabis with DI-SPME Arrow and LI-Syringe.\n\n\n\nHops pellets and cannabis flower preparation \nHops pellets were utilized as a terpene-free surrogate to matrix match cannabis flower for the following DI-SPME Arrow vs. LI syringe data: calibration curves, laboratory control samples (LCS), continuing calibration verification (CCV) samples, detection limit, and analytical precision samples. Hops were crushed and homogenized on a sheet pan with a rolling pin. The crushed hops were then cleaned with a proprietary solvent cleaning process to eliminate the presence of terpenes. Following solvent cleaning, the hops were dried in an oven. For the DI-SPME Arrow and LI syringe method precision experiments, cannabis shake (small pieces of cannabis flower that break off of larger buds) was homogenized and utilized. For the cannabis chemovar experiments, the flower was crushed and homogenized on a sheet pan using a rolling pin.\n\nAccelerated solvent extractor \nThe following DI-SPME Arrow vs. LI syringe experiments were conducted utilizing hops pellets and cannabis flowers, which were extracted using an ASE 350 with the parameters previously shown in Table 1. For all of the following DI-SPME Arrow and LI syringe experiments, either 0.5 g of cleaned hops or 0.5 g of cannabis flower was weighed out and placed into a 10 mL ASE 350 stainless steel extraction cell. Diatomaceous earth was then slowly added and lightly packed to fill the remaining volume in the cell. Samples were then extracted using IPA. Other work has shown that extracting with IPA can lead to poor peak shape for the terpenes of interest.[26] However, IPA gave desirable peak shape for this study and was used because of its cost, convenience, and toxicity relative to other solvents demonstrated for cannabis extractions.\n\nAfter ASE processing \nAfter ASE extraction, all extracts, which were typically between 10 and 11 mL, were brought to a final volume of 12 mL in order to consistently evaluate extracts of the same volume. Using a 3 mL Luer lock syringe with 0.22 \u00b5m filter, 3 mL of extract was filtered. For DI-SPME Arrow experiments, 1 mL of the filtered extract was added to 19 mL of LCMS grade water (i.e., 20 mL final volume) in a 20 mL headspace vial. In addition, 20 \u00b5L of 100 \u03bcg\/mL internal standard (ISTD) solution was added. Subsequently, the headspace vial was capped and spun for 10 seconds. For LI syringe experiments, 500 \u00b5L of the filtered extract was added to a 2 mL autosampler vial. In addition, 5 \u00b5L of the 10 \u03bcg\/mL ISTD solution was added. Subsequently, the autosampler vial was capped and spun for 10 seconds.\n\nTerpenes standards and internal standards \nDifferences in linear range between DI-SPME Arrow and LI syringe necessitated the use of the different intermediate and ISTD solutions. Intermediate concentrations of 1000 \u03bcg\/mL and a 10 \u03bcg\/mL were prepared from the 2,500 \u03bcg\/mL Terpene Standards 1 and 2. To prepare the 1000 \u03bcg\/mL intermediate, 400 \u00b5L of each terpene standard (i.e., 800 \u00b5L total) was added to 200 \u00b5L of IPA, then capped and vortexed. The 10 \u03bcg\/mL intermediate was prepared from the 1000 \u03bcg\/mL intermediate by adding 10 \u00b5L of the 1000 \u03bcg\/mL intermediate to 990 \u00b5L of IPA, then capped and vortexed. A solution of naphthalene-d8 ISTD was made at 100 \u03bcg\/mL for DI-SPME Arrow experiments and 10 \u03bcg\/mL for LI syringe experiments.\n\nDI-SPME Arrow calibration \nFor the highest DI-SPME Arrow calibration point (level 7), 153.6 \u00b5L of 1000 \u03bcg\/mL terpene solution was added to hops. Once extracted, the extract was brought to 12 mL with IPA and filtered, thereby reducing calibration level 7\u201312.8 \u03bcg\/mL. Intermediate serial dilutions (Supplementary Table S5) were carried out on calibration level 7 to make the other six calibration points. For example, 1500 \u00b5L of calibration level 7 was added to 1500 \u00b5L of IPA to make calibration level 6. This process was then repeated for the other calibration points. However, the final calibration solutions required a secondary dilution into 20 mL headspace vials (Supplementary Table S6). For example, 1 mL of the calibration level 6 (i.e., 6.4 \u03bcg\/mL) was added to 19 mL of water (i.e., 20 mL total volume) for a final concentration of 0.32 \u03bcg\/mL and then spiked with ISTD. For a DI-SPME Arrow CCV equivalent to calibration level 3, calibration 7 filtered extract was diluted with IPA, spiked with ISTD, capped, and vortexed for 10 seconds.\n\nLI syringe calibration \nFor the highest LI syringe calibration point (level 10), 61.4 \u00b5L of the 1000 \u03bcg\/mL terpene solution was added to the hops. Once extracted, the extract was brought to 12 mL with IPA and filtered, thereby reducing calibration level 10 to 5.12 \u03bcg\/mL. Serial dilutions (Supplementary Table S7) were carried out on calibration level 10 to make the other nine calibration points. For example, 500 \u00b5L of calibration level 10 was added to 500 \u00b5L of IPA to make calibration level 9. This process was then repeated for the other calibration points. Finally, 5 \u00b5L of the 10 \u03bcg\/mL ISTD solution was added to each calibration vial at levels 2\u201310, and 10 \u00b5L of the 10 \u03bcg\/mL ISTD solution was added to level 1 given the difference in final volume. After being spiked with ISTD, the vial was capped and spun for 10 seconds. See Supplementary Table S7 for the LI syringe calibration curve.\n\nMethod validation and chemovar experiments \nThis section addresses the following method validations: method detection limit (MDL)\/limit of quantitation (LOQ), analytical precision, method precision, and % recovery. The DI-SPME Arrow and LI syringe MDLs\/LOQs were determined from seven replicate low-level calibration points. In addition, LCSs were run to determine the analytical precision and % recovery of both methods. For a DI-SPME Arrow LCS, 76.8 \u00b5L of the 1000 \u03bcg\/mL intermediate terpene solution was added to hops (equivalent to calibration level 6). For an LI syringe LCS, 384 \u00b5L of 10 \u03bcg\/mL intermediate terpene solution was added to the hops (equivalent to calibration level 6). It is important to note that the LCS represents a separate hops spike and extraction. Furthermore, the DI-SPME Arrow and LI syringe method precisions were determined from seven different aliquots of cannabis shake. Finally, three different chemovars of cannabis flower were evaluated for terpenes with DI-SPME Arrow and LI syringe.\n\nResults and discussion \nHS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow \nInitial work compared three different types of sample preparation\/introduction techniques for terpene analysis via GC-MS. Techniques were evaluated based on relative compound response using only reference terpene standards. First, the more traditional approach using a HS syringe was compared to HS-SPME Arrow (120 \u00b5m DVB\/PDMS). As shown in Figure 1, 13 of 23 terpenes were identified using the HS syringe approach. However, this approach was unable to effectively pick up the later eluting and less volatile terpenes, which fall into the sesquiterpene category. When samples were analyzed via HS-SPME Arrow, 23 of 23 terpenes were able to be identified.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Analytical parameters for evaluating terpenes in cannabis with DI-SPME Arrow and LI-Syringe\n\n\n\nWhen comparing responses for the 13 terpenes that were able to be identified in both approaches, HS-SPME Arrow had much greater responses than the HS syringe approach. For the terpenes found in both HS techniques, the responses on the SPME Arrow were >10\u00d7 that of the HS syringe. Both samples were prepared identically and analyzed with the suggested parameters for each technique. When first looking at the HS syringe results, it was unclear if the less volatile sesquiterpenes were partitioning into the HS of the 20 mL vial. However, after analyzing the results for the HS-SPME Arrow and detecting the less volatile compounds, it was confirmed that these compounds are partitioning into the HS of the vial. It is not clear as to where the terpenes were lost (i.e., not transferred efficiently) in the HS syringe process, and it was outside of the scope of this study to determine the root cause. Because the HS-SPME Arrow method was able to identify all of the terpenes in the samples, this approach was chosen to move forward in the study. However, it was desired to see how HS-SPME Arrow compared to DI-SPME Arrow.\nHS-SPME Arrow samples and DI-SPME Arrow samples were prepared according to their respective approach, but they were analyzed under the same instrument conditions. Both techniques were able to identify all terpenes within the reference standard samples. However, as shown in Figure 2, terpene samples analyzed via DI-SPME Arrow showed improved responses over HS-SPME Arrow, especially for the higher molecular weight terpenes and also proved to be more reproducible (i.e., provide better precision). Responses for the DI-SPME Arrow averaged 6\u00d7 greater than that of the HS-SPME Arrow. %RSDs for the HS-SPME Arrow were as high as 76%, while all DI-SPME Arrow %RSDs were \u226415%. Potential limitations of DI-SPME Arrow include shortened fiber lifetime and\/or increased matrix exposure; however, given the improved responses and reproducibility, it was selected as the technique to move forward for further method validation and compared against a traditional liquid injection (LI) syringe method.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. HS-SPME Arrow vs DI-SPME Arrow for terpenes\n\n\n\nHand shakeout vs. accelerated solvent extractor \nSeveral terpene extraction approaches were considered for the current study. The full evaporative technique (FET), which is popular within the cannabis testing industry, was not evaluated in the current study as this technique\u2019s foundation is HS syringe. Additionally, the results discussed in HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow demonstrated that HS syringe did not perform as well as HS-SPME Arrow, which was also inferior to DI-SPME Arrow for the analysis of terpenes. Other industries already capitalize on the benefits of ASE 350[21][22][23][24] Therefore, a simple hand shakeout solvent extraction method was compared to an ASE 350 extraction method to evaluate the performance of each technique for extracting terpenes from cannabis flower. Three different cannabis chemovars were extracted using both techniques, and the average of their FID responses were determined (Table 4). Both techniques extracted the same 13 terpenes from the cannabis flower. On average, the hand shakeout responses were better than the ASE 350 responses for 11 of the 13 terpenes detected. Given the small sample size, a nonparametric Kruskal-Wallis test was completed to compare the averages and determine if there was a statistical difference between the hand shakeout and ASE 350 approaches. With the exception of camphene and linalool (p = 0.050), the Kruskal\u2013Wallis tests indicate a general trend of no statistically significant difference between the hand shakeout and ASE 350 extraction techniques for the 13 detected terpenes.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 4. Hand shakeout vs. accelerated solvent extraction (ASE 350) for extraction of terpenes from cannabis flower\n\n\n\nIn addition, several factors were considered when selecting the extraction method. The hand shakeout required more use of nonreusable consumables (e.g., centrifuge tubes), making it a \u201cgreener\u201d approach. In addition, the cannabis flower hand shakeout technique could potentially differ depending on the lab technician completing the manually intensive hand shakeout extraction. On the contrary, the ASE 350 cells were reused by cleaning after extraction. Furthermore, the extraction consistency that the ASE 350 offers was not user dependent. The lack of statistical significance between the two extraction techniques, coupled with less consumable needs and user variability, leads to utilizing the ASE 350 for all of the following DI-SPME Arrow vs. LI syringe experiments. It is important to note that this was the first study to utilize an ASE 350 for the extraction of terpenes from cannabis. Despite this novel development for the field of cannabis, future studies should consider the further development of the current ASE 350 parameters to optimize extraction efficiency by changing solvent extraction ratios, extraction temperature, etc. In addition, this technique lends itself very well to joint application potential with potency and\/or pesticides, therefore making it a very desirable technique.[27]\n\nHops for clean surrogate matrix \nAn initial comparison of identical calibration curves prepared in 100% IPA and matrix demonstrated matrix effects (MEs).[20] Of the 23 terpenes evaluated, 17 had positive MEs (seven average), as defined by Chambers et al.[28] This observation suggested there was a \u201csoft\u201d signal enhancement for these compounds, which may be the result of matrix-induced chromatographic response.[29] The remaining six terpenes (linalool, isopulegol, geraniol, \u03b2-caryophyllene, \u03b1-humulene, and trans-nerolidol) had negative MEs (-18 average), which suggested a \u201csoft\u201d signal suppression. The signal suppression for the aforementioned alcohols is contradictory to the theory of matrix-induced chromatographic response and represents a testament to the complexities of matrix effects. Regardless, these results indicated that cannabis flower MEs were present and therefore a matrix match calibration approach was deemed ideal. However, it was outside the scope of the current manuscript to fully dissect all of the current ME phenomenon associated with cannabis, especially considering the wide array of cannabis matrices. Future researchers are encouraged to expand upon the current start to understanding cannabis MEs.\nTo date, the cannabis industry has not been utilizing surrogate matrices, despite the fact that studies have demonstrated matrix effects could lead to inaccurate reporting.[20] Due to the numerous types of matrices cannabis testing laboratories analyze, laboratories are forced to become creative when doing matrix matching for their calibrations. Matrix matching for terpenes in cannabis flower represents a particularly tough issue, because similar plant species to cannabis also contain terpenes. In this study, a novel method of cleaning hops was utilized to provide a terpene-free surrogate for matrix matched calibrations. Matrix blanks were run to demonstrate the cleaned hops were free of terpenes and did not contribute to compound responses for the terpenes of interest (Figure 3). This was the first study to utilize cleaned hops as a clean surrogate for cannabis flower, and the following method validation results demonstrate not only that this was a viable technique, but also it produced a desirable outcome (i.e., excellent method performance).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3. Cleaned hops blank (LI syringe) demonstrating terpene-free surrogate matrix\n\n\n\nMethod performance (DI-SPME Arrow vs. LI syringe) \nMethod validation was done in accordance with the California Bureau of Cannabis Control guidelines and regulations.[30] Terpenes were analyzed via GC-MS\/MS using SIM in single quad MS mode. Single quad MS mode was used to be more representative of most cannabis laboratories, and SIM was used to minimize matrix interferences. All terpenes of interest were resolved using the Rxi-624Sil MS, 30 m \u00d7 0.25 mm \u00d7 1.4 \u00b5m (cat# 13868), as shown in Figure 4.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4. Full separation of 23 terpenes with GC-MS (LI syringe)\n\n\n\nDI-SPME Arrow terpene validation \nTerpene method validation was completed for DI-SPME Arrow to evaluate performance and possible implementation into cannabis testing laboratories. Method performance can be seen in Table 5. As shown in Table 5, an average r2 value of 0.991 was achieved for the terpenes of interest in an average working range of 0.08\u20130.64 \u03bcg\/mL. \u03b1-pinene had the lowest r2 value at 0.965. While most compounds had a working range of 0.08\u20130.64 \u03bcg\/mL for calibration, a few compounds (e.g., \u03b1-bisabolol) were able to have expanded ranges from 0.04\u20130.64 \u03bcg\/mL and 0.02\u20130.64 \u03bcg\/mL. However, two compounds (\u03b2-caryophyllene and \u03b1-humulene) had shortened ranges from 0.08 to 0.32 \u03bcg\/mL. The differences in calibration working ranges can be attributed to each individual compound\u2019s ability to ad\/absorb to the phase on the SPME Arrow and desorb, which can lead to the compound saturation. When compound saturation occurs within the calibration, the calibration curve will plateau, resulting in poor r2 values. Previous work looked at higher calibration ranges (>1 \u03bcg\/mL); however, because the Arrow phase reached its saturation point for this compound class, the calibration range needed to be lowered to a more appropriate linear range. This phenomenon can also be combated by adjusting the parameters (e.g., extraction time, desorption time, etc.), but changing these parameters could then lead to poor results for additional compounds.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 5. DI-SPME Arrow\u2013GC-MS method validation with ASE extracted cleaned hops matrix match for cannabis\n\n\n\nAn average LOQ of 0.050 \u03bcg\/mL was achieved for the DI-SPME Arrow method. The LOQ was three times the MDL, which was determined off of seven (n = 7) replicate injections at 0.08 \u03bcg\/mL. MDLs were calculated as the standard deviation of the seven replicate measurements multiplied by 3.14 (i.e., the Student\u2019s t-value for 99% confidence for seven values). Analytical precision was evaluated using reference standards. Seven replicate injections were made and %RSDs fell below 10% (average = 6.77%) for all terpenes of interest, proving that DI-SPME Arrow has the ability to be a robust analytical technique. In addition to analytical precision, overall method precision (i.e., ASE 350 and DI-SPME Arrow GC-MS) was evaluated by analyzing cannabis shake. Seven shake extractions were made and one sample from each extraction was analyzed to evaluate method precision. As seen in Table 5, there is a variance in %RSDs between the different terpenes of interest, and not all terpenes were detected in the shake. Overall, an average %RSD of 14.89% was achieved for the 17 terpenes detected in the cannabis shake. With 14 of the 17 detected terpenes having acceptable %RSDs (i.e., <30%), the remaining three terpenes (e.g., \u03b2-caryophyllene) having borderline unacceptable %RSDs. The % recoveries ranged from 67.2 to 262%, with an average of 95.5%. Three compounds (\u03b1-terpinene, terpinolene, and \u03b2-caryophyllene) had recoveries outside the acceptable \u00b130% window and were 262, 172, and 67.2%, respectively.\n\nLiquid injection syringe terpene validation \nMethod performance was evaluated for a LI syringe method and can be seen in Table 6. Over an average calibration working range of 0.04\u20135.12 \u03bcg\/mL, an average r2 of 0.993 was achieved for the 23 terpenes of interest. \u03b2-caryophyllene and \u03b1-bisabolol had the smallest calibration range spanning from 0.17 to 5.12 \u03bcg\/mL, but maintained good r2 values of 0.992 and 0.990, respectively. Twenty-one of the 23 terpenes of interest had an r2 value of \u22650.990, while the additional two compounds (geraniol and trans-nerolidol) had an r2 value of 0.988. It is important to note that the LI syringe technique was able to achieve a higher range for the terpenes.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 6. LI syringe\u2013GC-MS method validation with ASE extracted cleaned hops matrix for cannabis\n\n\n\nAn average LOQ of 0.047 \u03bcg\/mL was achieved for the LI syringe method, which is nearly identical to the average LOQ achieved for the DI-SPME Arrow method. LOQs for the LI syringe method were calculated identically to what was completed to those for the DI-SPME Arrow method. It should be noted that while the LI syringe method had lower LOQs for the more volatile terpenes of interest, the DI-SPME Arrow method had lower LOQs for the majority of the less volatile terpenes of interest. Analytical precision had an average %RSD of 1.62%, with all compounds below 5%. Improvements were made when looking at method precision via LI syringe vs. DI-SPME Arrow. All terpenes of interest had %RSDs of less than 10%, with the exception of cis-nerolidol (<15%) for method precision. Average %RSD for LI syringe was 4.75%. The % recoveries ranged from 84.6 to 98.9%, with an average of 90.2%. All 23 terpenes of interest fell into the acceptable \u00b130% window for the LI syringe method.\nThe aforementioned results are comparable to a previously published study by Ibrahim et al.[12], which showed r2 value of >0.99, %RSDs <15%, recoveries ranging from 67 to 105.7%, MDLs of 0.25 \u03bcg\/mL, and LOQs of 0.75 \u03bcg\/mL. The current study was able to build off of the foundation from Ibrahim et al. and achieved similar r2 values, %RSDs, and recoveries. However, this study excelled with lower working calibration range and an order of magnitude lower LOQs (0.047 \u03bcg\/mL average). In addition, this study employed the use of accelerated solvent extraction and matrix matched calibrations.\n\nTerpene chemovar evaluation with DI-SPME Arrow and LI syringe \nAfter method validation, the chemical variety (chemovar) of three different cannabis chemovars was evaluated via ASE and both DI-SPME Arrow and LI syringe techniques. Results shown in Table 7 represent the concentration in the sample for Mint Chocolate Chip (i.e., corrected for ASE extraction and dilution). It is important to note that initially three different chemovars (n = 9 total) were evaluated by both DI-SPME Arrow and LI syringe; however, due to SPME Arrow saturation on the first two chemovars, the Mint Chocolate Chip chemovar underwent a secondary 10\u00d7 dilution of the cannabis extract. From the data collected, several points should be discussed when comparing the two methods. For the DI-SPME Arrow method, 17 of 23 terpenes were identified and 22 of 23 were identified via LI syringe. It should be noted that one compound (p-cymene) was not detected (ND) by either technique and should be considered as not present in this chemovar. The DI-SPME Arrow method had several compounds falling outside of the working ranges and several compounds that were ND. It is hypothesized that perhaps the SPME Arrow used for this experiment was reaching the end of its viable lifetime and therefore was not as efficiently adsorbing and\/or releasing the terpenes; however, this was not determined. Regardless, the LI syringe appeared to provide better results when analyzing a true sample. Nearly all compounds fell within the LI syringe calibration working ranges and saturation was not an issue.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 7. Mint Chocolate Chip terpene content determined by DI-SPME Arrow and LI syringe\n\n\n\nDespite the aforementioned discrepancies, the average concentration for DI-SPME Arrow was 18.3 \u03bcg\/g, while the LI syringe average concentration was 16.7 \u03bcg\/g. Comparing the average concentrations specifically, both techniques showed similar concentrations for half of the compounds and divergent concentrations for the other half. More importantly, when comparing the overall terpene profiles of each injection technique, the profiles showed similar results (Figure 5). In particular, both DI-SPME Arrow and LI syringe shared the same top five most abundant compounds, which include \u03b1-pinene, \u03b2-pinene, D-limonene, linalool, and \u03b2-caryophyllene. Granted, \u03b1-pinene\u2019s response was almost 2\u00d7 greater for DI-SPME Arrow, while \u03b2-caryophyllene\u2019s response was over 2.5\u00d7 greater for LI syringe. Alternatively, the percentages for D-limonene and linalool fell within 3% of one another. Note that the other 18 terpenes were summed in the \u201cother\u201d category. The LI syringe had more positive hits for lower concentration terpenes, as its calibration range was lower than the DI-SPME Arrow, hence having the 29% \u201cother\u201d for LI syringe vs. the 15% \u201cother\u201d for DI-SPME Arrow.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5. Mint Chocolate Chip terpene profiles determined by DI-SPME Arrow and LI syringe\n\n\n\nConclusions, future work, and recommendations \nThe current study demonstrated that DI-SPME Arrow performed better than HS-SPME Arrow; however, both of these approaches outperformed HS syringe for the extraction and analyses of terpenes. A novel method for cleaning hops was developed to provide terpene-free hops utilized as a surrogate for matrix matched calibrations. For the first time, both hops and cannabis flower were extracted with ASE and then utilized for method validation.\nValidated methodologies for analyzing terpenes in cannabis flower were developed for both DI-SPME Arrow and liquid injection syringe with GC-MS\/MS in single quad mode and selected ion monitoring. Both methods proved to be viable options for the analysis of terpenes in cannabis flower. When comparing the average LOQs of the terpenes of interest, both techniques were near identical. However, results suggest that LI syringe would be the preferred approach for this analysis based on several observations. The LI syringe showed an expanded working calibration range compared to the DI-SPME Arrow, which can be attributed to phase saturation of the SPME Arrow. In addition, better analytical and method precision was achieved by LI syringe. Furthermore, LI syringe appeared to provide a more complete chemovar profile of cannabis flower and at higher concentrations. Lastly, SPME Arrows needed routine replacement due to phase swelling and\/or lifetime, a fate which was not shared by LI syringe.\nWhile results indicate that LI syringe is the preferred technique, other factors should be considered for future work. When running LI syringe, more GC maintenance may be needed when compared to the popular HS methods in the cannabis testing industry. This includes inlet consumable changes, analytical column trimming, and MS maintenance, which will come as a result of the matrix being injected into the system. One area not explored in the current work that should be evaluated in future work is instrument uptime for DI-SPME Arrow vs. LI syringe. It is hypothesized that less matrix may be exposed to the GC when running the DI-SPME Arrow method, which in turn may lead to longer instrument uptime and less time and money spent on maintenance.\nIt is recommended that the scientific cannabis community reconsider utilizing HS-Syringe for the analysis of terpenes in cannabis products, as the current results suggest it is inferior to all of the approaches discussed, especially LI syringe. Therefore, it is also recommend that the full evaporative technique, which is an HS syringe-based approach, also be reassessed in the future. Furthermore, the FET approach is not amenable to splitting samples for other cannabis test methods (e.g., potency), like the ASE method outlined in this study. In addition, it is recommended that future cannabis work continue to evaluate the use of terpene-free hops for surrogate matrix matching of flower. Furthermore, it is recommended that additional cannabis research further the development of ASE methods for the extraction and analysis of terpenes. Under the appropriate conditions, the ASE has the potential to improve laboratory workflows by using one extraction and splitting that extract between multiple analyses in addition to terpene profiling (e.g., potency, pesticides, and mycotoxins). Lastly, it is recommended that this work should be expanded to additional matrices that cannabis testing laboratories frequently analyze (e.g., shatters and waxes), and expand upon the cannabis ME knowledge, which has only begun here. These additional matrices bring new challenges and will need to be addressed to improve the science of cannabis testing.\n\nSupplementary material \n Datasheet 1 (PDF)\n Datasheet 2 (PDF)\nAcknowledgements \nAuthor contributions \nCM started this study by investigating different approaches for analyzing terpenes. CM and JH outlined the research plan, analyzed data, and wrote the manuscript. PH facilitated the research by providing a licensed cannabis testing laboratory to continue the research. KA conducted the experiments outlined by CM and JH at the laboratory owned by PH.\n\nData availability statement \nThe original contributions presented in the study are included in the article\/Supplementary Material; further inquiries can be directed to the corresponding author.\n\nConflict of interest \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 Stelton-Holtmeier, J. (25 August 2020). \"Chart: Nationwide sales of adult-use cannabis further eclipse those of medical marijuana\". Marijuana Business Daily. https:\/\/mjbizdaily.com\/chart-nationwide-sales-of-adult-use-cannabis-further-eclipse-those-of-medical-marijuana\/ . Retrieved 31 August 2020 .   \n\n\u2191 2.0 2.1 \"Chart: Nationwide sales of adult-use cannabis further eclipse those of medical marijuana\". February 2020. https:\/\/www.marketdataforecast.com\/market-reports\/cannabis-testing-market . Retrieved 31 August 2020 .   \n\n\u2191 3.0 3.1 Nuutinen, T. (2018). \"Medicinal properties of terpenes found in Cannabis sativa and Humulus lupulus\". European Journal of Medicinal Chemistry 157: 198\u2013228. doi:10.1016\/j.ejmech.2018.07.076. PMID 30096653.   \n\n\u2191 4.0 4.1 4.2 4.3 Calvi, L.; Pentimalli, D.; Panseri, S. et al. (2018). \"Comprehensive quality evaluation of medical Cannabis sativa L. inflorescence and macerated oils based on HS-SPME coupled to GC\u2013MS and LC-HRMS (q-exactive orbitrap) approach\". Journal of Pharmaceutical and Biomedical Analysis 150: 208\u201319. doi:10.1016\/j.jpba.2017.11.073.   \n\n\u2191 Russo, E.B. (2011). \"Taming THC: Potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\". British Journal of Pharmacology 163 (7): 1344-64. doi:10.1111\/j.1476-5381.2011.01238.x. PMC PMC3165946. PMID 21749363. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946 .   \n\n\u2191 Abilleira, E.; de Renobales, M.; N\u00e1jera, A.I. et al. (2010). \"An accurate quantitative method for the analysis of terpenes in milk fat by headspace solid-phase microextraction coupled to gas chromatography\u2013mass spectrometry\". Food Chemistry 120 (4): 1162-1169. doi:10.1016\/j.foodchem.2009.11.050.   \n\n\u2191 Kupska, M.; Chmiel, T.; J\u0119drkiewicz, R. et al. (2014). \"Comprehensive two-dimensional gas chromatography for determination of the terpenes profile of blue honeysuckle berries\". Food Chemistry 152: 88\u201393. doi:10.1016\/j.foodchem.2013.11.129. PMID 24444910.   \n\n\u2191 Cacho, J.I.; Campillo, N.; Vi\u00f1as, P. et al. (2015). \"Evaluation of three headspace sorptive extraction coatings for the determination of volatile terpenes in honey using gas chromatography-mass spectrometry\". Journal of Chromatography A 1399: 18\u201324. doi:10.1016\/j.chroma.2015.04.041. PMID 25958092.   \n\n\u2191 Jele\u0144, H.H.; Gracka, A. (2015). \"Analysis of black pepper volatiles by solid phase microextraction-gas chromatography: A comparison of terpenes profiles with hydrodistillation\". Journal of Chromatography A 1418: 200\u2013209. doi:10.1016\/j.chroma.2015.09.065. PMID 26427328.   \n\n\u2191 10.0 10.1 10.2 Stenerson, K.K.; Halpenny, M.R. (2017). \"Analysis of Terpenes in Cannabis Using Headspace Solid-Phase Microextraction and GC\u2013MS\". LCGC 35 (5): 27\u201332. https:\/\/www.chromatographyonline.com\/view\/analysis-terpenes-cannabis-using-headspace-solid-phase-microextraction-and-gc-ms .   \n\n\u2191 11.0 11.1 11.2 Brown, A.K.; Xia, Z.; Bulloch, P. et al. (2019). \"Validated quantitative cannabis profiling for Canadian regulatory compliance - Cannabinoids, aflatoxins, and terpenes\". Analytica Chimica Acta 1088: 79\u201388. doi:10.1016\/j.aca.2019.08.042. PMID 31623719.   \n\n\u2191 12.0 12.1 12.2 12.3 Ibrahim, E.A.; Wang, M.; Radwan, M.M. et al. (2019). \"Analysis of Terpenes in Cannabis sativa L. Using GC\/MS: Method Development, Validation, and Application\". Planta Medica 85 (5): 431\u201338. doi:10.1055\/a-0828-8387. PMID 30646402.   \n\n\u2191 13.0 13.1 13.2 Shapira, A.; Berman, P. Futoran, K. et al. (2019). \"Tandem Mass Spectrometric Quantification of 93 Terpenoids in Cannabis Using Static Headspace Injections\". Analytical Chemistry 91 (17): 11425\u201332. doi:10.1021\/acs.analchem.9b02844. PMID 31369251.   \n\n\u2191 14.0 14.1 14.2 Bakro, F.; Jedryczka, M.; Wielgusz, K. et al. (2020). \"Simultaneous determination of terpenes and cannabidiol in hemp (Cannabis sativa L.) by fast gas chromatography with flame ionization detection\". Journal of Separation Science 43 (14): 2817\u201326. doi:10.1002\/jssc.201900822.   \n\n\u2191 15.0 15.1 15.2 Gaggiotti, S.; Palmieri, S.; Pelle, F.D. et al. (2020). \"Piezoelectric peptide-hpDNA based electronic nose for the detection of terpenes; Evaluation of the aroma profile in different Cannabis sativa L. (hemp) samples\". Sensors and Actuators B: Chemical 308: 127697. doi:10.1016\/j.snb.2020.127697.   \n\n\u2191 Mu\u00f1oz-Redondo, J.M.; Ruiz-Moreno, M.J.; Puertas, B. et al. (2020). \"Multivariate optimization of headspace solid-phase microextraction coupled to gas chromatography-mass spectrometry for the analysis of terpenoids in sparkling wines\". Talanta 208: 120483. doi:110.1016\/j.talanta.2019.120483.   \n\n\u2191 17.0 17.1 Nguyen, T.-D.; Riordan-Short, S.; Dang, T.-T.T. et al. (2020). \"Quantitation of Select Terpenes\/Terpenoids and Nicotine Using Gas Chromatography-Mass Spectrometry with High-Temperature Headspace Sampling\". ACS Omega 5 (10): 5565-5573. doi:10.1021\/acsomega.0c00384. PMC PMC7081649. PMID 32201850. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7081649 .   \n\n\u2191 18.0 18.1 18.2 Ternelli, M.; Brighenti, V.; Anceschi, L. et al. (2020). \"Innovative methods for the preparation of medical Cannabis oils with a high content of both cannabinoids and terpenes\". Journal of Pharmaceutical and Biomedical Analysis 186: 113296. doi:10.1016\/j.jpba.2020.113296. PMID 32334134.   \n\n\u2191 Zhang, X.; Wang, C.; Wang, L. et al. (2020). \"Optimization and validation of a head space solid-phase microextraction-arrow gas chromatography-mass spectrometry method using central composite design for determination of aroma compounds in Chinese liquor (Baijiu)\". Journal of Chromatography A 1610: 460584. doi:10.1016\/j.chroma.2019.460584. PMID 31607446.   \n\n\u2191 20.0 20.1 20.2 Raposo, F.; Barcel\u00f3, D. (2021). \"Challenges and strategies of matrix effects using chromatography-mass spectrometry: An overview from research versus regulatory viewpoints\". TrAC Trends in Analytical Chemistry 134: 116068. doi:10.1016\/j.trac.2020.116068.   \n\n\u2191 21.0 21.1 Ligor, M.; Stankevi\u010dius, M.; Wenda-Piesik, A. et al. (2014). \"Comparative Gas Chromatographic\u2013Mass Spectrometric Evaluation of Hop (Humulus lupulus L.) Essential Oils and Extracts Obtained Using Different Sample Preparation Methods\". Food Analytical Methods 7: 1433\u201342. doi:10.1007\/s12161-013-9767-5.   \n\n\u2191 22.0 22.1 Chiesa, L.M.; Labella, G.F.; Panseri, S. et al. (2017). \"Accelerated solvent extraction by using an 'in-line' clean-up approach for multiresidue analysis of pesticides in organic honey\". Food Additives & Contaminants: Part A 34 (5): 809\u201318. doi:10.1080\/19440049.2017.1292558. PMID 28277176.   \n\n\u2191 23.0 23.1 Hu, A.; Qiu, M.; Liu, H. et al. (2020). \"Simultaneous determination of phthalate diesters and monoesters in soil using accelerated solvent extraction and ultra-performance liquid chromatography coupled with tandem mass spectrometry\". Journal of Chromatography A 1626: 104366. doi:10.1016\/j.microc.2019.104366.   \n\n\u2191 24.0 24.1 Min, N.; Yao, J.; Amde, M. et al. (2020). \"Accelerated solvent extraction combined with GC\u2013MS: A convenient technique for the determination and compound-specific stable isotope analysis of phthalates in mine tailings\". Microchemical Journal 153: 461347. doi:10.1016\/j.chroma.2020.461347. PMID 32797827.   \n\n\u2191 Herrington, J.S.; G\u00f3mez-R\u00edos, G.A.; Myers, C. et al. (2020). \"Hunting Molecules in Complex Matrices with SPME Arrows: A Review\". Separations 7 (1): 12. doi:10.3390\/separations7010012.   \n\n\u2191 Krill, C.; Rochfort, S.; Spangenberg, G. (2020). \"A High-Throughput Method for the Comprehensive Analysis of Terpenes and Terpenoids in Medicinal Cannabis Biomass\". Metabolites 10 (7): 276. doi:10.3390\/metabo10070276.   \n\n\u2191 \"Fast, Low-Solvent Analysis of Cannabinoids Increases Lab Productivity and Decreases Solvent Costs\". Restek. 2020. https:\/\/www.restek.com\/en\/technical-literature-library\/articles\/fast-low-solvent-analysis-of-cannabinoids-increases-lab-productivity-and-decreases-solvent-costs\/ . Retrieved 27 April 2021 .   \n\n\u2191 Chambers, E.; Wagrowski-Diehl, D.M.; Lu, Z. et al. (2007). \"Systematic and comprehensive strategy for reducing matrix effects in LC\/MS\/MS analyses\". Journal of Chromatography B 852 (1\u20132): 22\u201334. doi:10.1016\/j.jchromb.2006.12.030.   \n\n\u2191 Hajslov\u00e1, J.; Zrostl\u00edkov\u00e1, J. (2003). \"Matrix effects in (ultra)trace analysis of pesticide residues in food and biotic matrices\". Journal of Chromatography A 1000 (1\u20132): 181\u201397. doi:10.1016\/s0021-9673(03)00539-9. PMID 12877171.   \n\n\u2191 \"California Code of Regulations Title 16, Division 42, Bureau of Cannabis Control\" (PDF). Bureau of Cannabis Control. 2019. https:\/\/bcc.ca.gov\/law_regs\/cannabis_order_of_adoption.pdf . Retrieved 01 August 2020 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; this wiki organizes them by order of appearance, by design.\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\">https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis<\/a>\n\t\t\t\t\tCategories: CannaQAwiki journal articles (added in 2021)CannaQAwiki journal articles (all)CannaQAwiki journal articles on cannabis researchCannaQAwiki journal articles on cannabis testing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tList of articles\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 April 2021, at 16:24.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout CannaQAWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t","a3b638adc3dd9824430235085d4b072b_html":"<body class=\"mediawiki ltr sitedir-ltr ns-170 ns-subject page-Journal_Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis rootpage-Journal_Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Accelerated solvent extraction of terpenes in cannabis coupled with various injection techniques for GC-MS analysis<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>The <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis\" title=\"Cannabis\" class=\"wiki-link\" data-key=\"a70b76268930d795518ff1f98d7e500d\">cannabis<\/a> market is expanding exponentially in the United States. As state-wide <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Legality_of_cannabis\" title=\"Legality of cannabis\" class=\"wiki-link\" data-key=\"b80036537100734fce629cc441653451\">legalization<\/a> efforts increase, so also do demands for analytical testing methodologies. One of the main tests conducted on cannabis products is the analysis for <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Terpene\" title=\"Terpene\" class=\"wiki-link\" data-key=\"6bcbb95d582c08073101e9c4cfc91931\">terpenes<\/a>. This research focused on implementation of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Accelerated_solvent_extraction\" title=\"Accelerated solvent extraction\" class=\"wiki-link\" data-key=\"b19bbd6e8c42da7054bf6934b125e228\">accelerated solvent extraction<\/a> (ASE), utilizing surrogate matrix matching, and evaluation of traditional vs. more modern <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"2d7596b7c3616db8f56af698be478d3d\">sample<\/a> introduction techniques for analyzing terpenes via <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Gas_chromatography%E2%80%93mass_spectrometry\" title=\"Gas chromatography\u2013mass spectrometry\" class=\"wiki-link\" data-key=\"afbcab5867e499fab32b49e2b77c9e32\">gas chromatography\u2013mass spectrometry<\/a> (GC-MS). Introduction techniques included headspace syringe (HS syringe), HS-<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Solid-phase_microextraction\" title=\"Solid-phase microextraction\" class=\"wiki-link\" data-key=\"c0044f33396758ceb031c7c223ff6ddd\">solid-phase microextraction<\/a> Arrow (HS-SPME Arrow), direct immersion-SPME Arrow (DI-SPME Arrow), and liquid injection syringe (LI syringe). The LI syringe approach was deemed the most straightforward and robust method, with terpene working ranges of 0.04\u20135.12 \u03bcg\/mL; <i>r<\/i><sup>2<\/sup> values of 0.988\u20130.996 (0.993 average); limit of quantitation values of 0.017\u20130.129 \u03bcg\/mL (0.047 average); analytical precisions of 2.58\u20139.64% RSD (1.56 average); overall ASE-LI-syringe-GC-MS method precisions of 1.73\u201314.6% RSD (4.97 average); and % recoveries of 84.6\u201398.9% (90.2 average) for the 23 terpenes of interest. Sample <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"c63a2fc4c276562dbdaf957f640d69cb\">workflows<\/a> and results are discussed, with an evaluation of the advantages\/limitations of each approach and opportunities for future work.\n<\/p><p><b>Keywords<\/b>: accelerated solvent extraction (ASE), terpenes, solid-phase microextraction (SPME), solid-phase microextraction Arrow (SPME Arrow), gas chromatography\u2013mass spectrometry (GC-MS)\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>The legal <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis\" title=\"Cannabis\" class=\"wiki-link\" data-key=\"a70b76268930d795518ff1f98d7e500d\">cannabis<\/a> market is one of the fastest growing markets across the globe. In 2019, cannabis use for <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_(drug)\" title=\"Cannabis (drug)\" class=\"wiki-link\" data-key=\"6500790ffdd84957d4b055c47e48efca\">medicinal<\/a> purposes in the United States generated $4 billion to $4.9 billion in sales, compared to the adult-use estimates between $6.6 billion and $8.1 billion.<sup id=\"rdp-ebb-cite_ref-Stelton-HoltmeierChart20_1-0\" class=\"reference\"><a href=\"#cite_note-Stelton-HoltmeierChart20-1\">[1]<\/a><\/sup> As the United States and additional countries continue to <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Legality_of_cannabis\" title=\"Legality of cannabis\" class=\"wiki-link\" data-key=\"b80036537100734fce629cc441653451\">legalize<\/a> the use of medicinal and recreational cannabis, analytical testing demands increase. A 2020 report by Market Data Forecast valued the global cannabis testing market at $1,218.0 million in 2019 and estimated it to be growing at a compound annual growth rate (CAGR) of 12.42%.<sup id=\"rdp-ebb-cite_ref-MDFGlobal20_2-0\" class=\"reference\"><a href=\"#cite_note-MDFGlobal20-2\">[2]<\/a><\/sup> The market is projected to almost double at $2,187.3 million by 2024.<sup id=\"rdp-ebb-cite_ref-MDFGlobal20_2-1\" class=\"reference\"><a href=\"#cite_note-MDFGlobal20-2\">[2]<\/a><\/sup> Of the examinations conducted in cannabis testing <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"ed14e6a67b4b14ad2c190c28455725f6\">laboratories<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Terpene\" title=\"Terpene\" class=\"wiki-link\" data-key=\"6bcbb95d582c08073101e9c4cfc91931\">terpene<\/a> profiling is a popular analysis, regardless of state regulations. \n<\/p><p>Terpenes are a naturally occurring set of organic compounds, which are commonly found in plants, and are typically strong in odor.<sup id=\"rdp-ebb-cite_ref-NuutinenMedicinal18_3-0\" class=\"reference\"><a href=\"#cite_note-NuutinenMedicinal18-3\">[3]<\/a><\/sup> Terpenes are made up of isoprene units and are classified by the number of their isoprene units.<sup id=\"rdp-ebb-cite_ref-NuutinenMedicinal18_3-1\" class=\"reference\"><a href=\"#cite_note-NuutinenMedicinal18-3\">[3]<\/a><\/sup> The two types of terpenes that are commonly analyzed in the cannabis testing industry are monoterpenes, which have two isoprene units, and sesquiterpenes, which have three isoprene units. Over 100 terpenes have been identified in different cannabis chemical varieties (chemovars).<sup id=\"rdp-ebb-cite_ref-CalviCompre18_4-0\" class=\"reference\"><a href=\"#cite_note-CalviCompre18-4\">[4]<\/a><\/sup> Each cannabis chemovar has its own unique terpene profile, giving consumers different aromas, flavors, and experiences depending on the chemovar they use. According to Russo <i>et al.<\/i>, terpenes play a major role in the <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Entourage_effect\" title=\"Entourage effect\" class=\"wiki-link\" data-key=\"055e82f0e8d889abcab91887eaa36be5\">entourage effect<\/a>, which is the synergistic interaction between phytocannabinoids and terpenoids with respect to treating numerous ailments (e.g., depression).<sup id=\"rdp-ebb-cite_ref-RussoTaming11_5-0\" class=\"reference\"><a href=\"#cite_note-RussoTaming11-5\">[5]<\/a><\/sup> The desire to understand and capitalize on this entourage effect is the motivation for terpene testing in the cannabis industry.\n<\/p><p>Terpenes have been analyzed in numerous commodities within the food and beverage industry. Previous studies have looked at a variety of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Matrix_(chemical_analysis)\" title=\"Matrix (chemical analysis)\" class=\"wiki-link\" data-key=\"f13c807ddc78c669e419ec767efe4c7d\">matrices<\/a> (e.g., tequila) and have used different analytical techniques (e.g., <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Solid-phase_microextraction\" title=\"Solid-phase microextraction\" class=\"wiki-link\" data-key=\"c0044f33396758ceb031c7c223ff6ddd\">solid-phase microextraction<\/a> [SPME]) to conduct the analysis.<sup id=\"rdp-ebb-cite_ref-CalviCompre18_4-1\" class=\"reference\"><a href=\"#cite_note-CalviCompre18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AbilleiraAnAccur10_6-0\" class=\"reference\"><a href=\"#cite_note-AbilleiraAnAccur10-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KupskaComp14_7-0\" class=\"reference\"><a href=\"#cite_note-KupskaComp14-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CachoEval15_8-0\" class=\"reference\"><a href=\"#cite_note-CachoEval15-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Jele.C5.84Anal15_9-0\" class=\"reference\"><a href=\"#cite_note-Jele.C5.84Anal15-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-StenersonAnal17_10-0\" class=\"reference\"><a href=\"#cite_note-StenersonAnal17-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BrownValid19_11-0\" class=\"reference\"><a href=\"#cite_note-BrownValid19-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-IbrahimAnal19_12-0\" class=\"reference\"><a href=\"#cite_note-IbrahimAnal19-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShapiraTandem19_13-0\" class=\"reference\"><a href=\"#cite_note-ShapiraTandem19-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BakroSimul20_14-0\" class=\"reference\"><a href=\"#cite_note-BakroSimul20-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GaggiottiPiezoelectric20_15-0\" class=\"reference\"><a href=\"#cite_note-GaggiottiPiezoelectric20-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Mu.C3.B1oz-RedondoMulti20_16-0\" class=\"reference\"><a href=\"#cite_note-Mu.C3.B1oz-RedondoMulti20-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NguyenQuant20_17-0\" class=\"reference\"><a href=\"#cite_note-NguyenQuant20-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TernelliInnov20_18-0\" class=\"reference\"><a href=\"#cite_note-TernelliInnov20-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZhangOptim20_19-0\" class=\"reference\"><a href=\"#cite_note-ZhangOptim20-19\">[19]<\/a><\/sup> However, only a few studies have shown the analysis of terpenes in cannabis and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Hemp\" title=\"Hemp\" class=\"wiki-link\" data-key=\"c23e30b6cf1df54f1dc338492c9f9da2\">hemp<\/a> matrices (e.g., flower, gummy), and their robustness for compliance laboratories remains uncertain. Calvi <i>et al.<\/i>, Ternelli <i>et al.<\/i>, Gaggotti <i>et al.<\/i>, and Stenerson <i>et al.<\/i> did not perform extractions on cannabis and hemp samples; rather, they added the samples directly to a headspace (HS) vial and demonstrated the analysis of terpenes using HS-SPME.<sup id=\"rdp-ebb-cite_ref-CalviCompre18_4-2\" class=\"reference\"><a href=\"#cite_note-CalviCompre18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-StenersonAnal17_10-1\" class=\"reference\"><a href=\"#cite_note-StenersonAnal17-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GaggiottiPiezoelectric20_15-1\" class=\"reference\"><a href=\"#cite_note-GaggiottiPiezoelectric20-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TernelliInnov20_18-1\" class=\"reference\"><a href=\"#cite_note-TernelliInnov20-18\">[18]<\/a><\/sup> Nguyen <i>et al.<\/i> utilized a pseudo extraction by adding a solvent to dried material, followed by analysis via headspace <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Gas_chromatography%E2%80%93mass_spectrometry\" title=\"Gas chromatography\u2013mass spectrometry\" class=\"wiki-link\" data-key=\"afbcab5867e499fab32b49e2b77c9e32\">gas chromatography\u2013mass spectrometry<\/a> (HS-GC-MS).<sup id=\"rdp-ebb-cite_ref-NguyenQuant20_17-1\" class=\"reference\"><a href=\"#cite_note-NguyenQuant20-17\">[17]<\/a><\/sup> The five aforementioned studies appear to lack an exhaustive cannabis or hemp extraction, and therefore this calls into question the real-world applicability of these methods. Furthermore, Calvi <i>et al.<\/i>, Ternelli <i>et al.<\/i>, Gaggotti <i>et al.<\/i>, and Stenerson <i>et al.<\/i> only focused on profiling the terpenes in the cannabis or hemp matrices studied and therefore only presented qualitative and semi-quantitative data.<sup id=\"rdp-ebb-cite_ref-CalviCompre18_4-3\" class=\"reference\"><a href=\"#cite_note-CalviCompre18-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-StenersonAnal17_10-2\" class=\"reference\"><a href=\"#cite_note-StenersonAnal17-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GaggiottiPiezoelectric20_15-2\" class=\"reference\"><a href=\"#cite_note-GaggiottiPiezoelectric20-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TernelliInnov20_18-2\" class=\"reference\"><a href=\"#cite_note-TernelliInnov20-18\">[18]<\/a><\/sup>\n<\/p><p>Bakro <i>et al.<\/i>, Brown <i>et al.<\/i>, Ibrahim <i>et al.<\/i>, and Shapira <i>et al.<\/i> extracted cannabis <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Inflorescence\" title=\"Inflorescence\" class=\"wiki-link\" data-key=\"d8122354225ee6a932a086ca8a1b999f\">flower<\/a> with <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Ethanol\" title=\"Ethanol\" class=\"wiki-link\" data-key=\"e0bdbbae89fcedde33e133fdfd2a2994\">ethanol<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Hexane\" title=\"Hexane\" class=\"wiki-link\" data-key=\"f505fabbdca1dd9cd6f3c163138ef7a6\">hexane<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Ethyl_acetate\" title=\"Ethyl acetate\" class=\"wiki-link\" data-key=\"11761f9cd1ab8b5718334c9a9feacd79\">ethyl acetate<\/a>, and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Methanol\" title=\"Methanol\" class=\"wiki-link\" data-key=\"e0e6fe7bacded0b162958a69f290a7bd\">methanol<\/a>, respectively, and provided quantitative results.<sup id=\"rdp-ebb-cite_ref-BrownValid19_11-1\" class=\"reference\"><a href=\"#cite_note-BrownValid19-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-IbrahimAnal19_12-1\" class=\"reference\"><a href=\"#cite_note-IbrahimAnal19-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShapiraTandem19_13-1\" class=\"reference\"><a href=\"#cite_note-ShapiraTandem19-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BakroSimul20_14-1\" class=\"reference\"><a href=\"#cite_note-BakroSimul20-14\">[14]<\/a><\/sup> However, Bakro <i>et al.<\/i> only looked at hemp and used a nonspecific gas chromatography with flame-ionization detection (GC-FID) approach, which is cumbersome when attempting to differentiate between coeluting terpenes of interest and matrix interferences.<sup id=\"rdp-ebb-cite_ref-BakroSimul20_14-2\" class=\"reference\"><a href=\"#cite_note-BakroSimul20-14\">[14]<\/a><\/sup> Brown <i>et al.<\/i> did not provide method accuracies for all targeted terpenes and reported less than desirable linearities, which fell below an <i>r<\/i><sup>2<\/sup> value of 0.960 for each terpene of interest.<sup id=\"rdp-ebb-cite_ref-BrownValid19_11-2\" class=\"reference\"><a href=\"#cite_note-BrownValid19-11\">[11]<\/a><\/sup>\n<\/p><p>To date, the most promising methods\u2014presented by Ibrahim <i>et al.<\/i> and Shapira <i>et al.<\/i>\u2014utilize exhaustive cannabis and hemp extraction approaches, followed by GC-MS and reported desirable quantitative results.<sup id=\"rdp-ebb-cite_ref-IbrahimAnal19_12-2\" class=\"reference\"><a href=\"#cite_note-IbrahimAnal19-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShapiraTandem19_13-2\" class=\"reference\"><a href=\"#cite_note-ShapiraTandem19-13\">[13]<\/a><\/sup> Ibrahim <i>et al.<\/i> and Shapira <i>et al.<\/i> used sample introduction techniques like liquid injection without filtration and static headspace GC-MS (SHS-GC-MS), respectively. More importantly, none of the aforementioned studies accounted for matrix effects, as they all used solvent-based calibrations and, due to the complexity and dirtiness of cannabis matrices, this could lead to inaccurate reporting.<sup id=\"rdp-ebb-cite_ref-RaposoChall21_20-0\" class=\"reference\"><a href=\"#cite_note-RaposoChall21-20\">[20]<\/a><\/sup> In addition, these studies did not evaluate more modern sample extraction approaches [e.g., <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Accelerated_solvent_extraction\" title=\"Accelerated solvent extraction\" class=\"wiki-link\" data-key=\"b19bbd6e8c42da7054bf6934b125e228\">accelerated solvent extraction<\/a> (ASE)] or sample introduction techniques (e.g., direct immersion-SPME Arrow [DI-SPME Arrow]).\n<\/p><p>The following study was conducted to evaluate more modern sample preparation and introduction techniques and demonstrate their potential value to cannabis compliance testing laboratories in need of guidance for qualitative and quantitative terpenes analysis. In addition, this study evaluated accelerated solvent extraction (ASE 350) of terpenes in cannabis samples, which is commonly used in other markets within the analytical testing industry.<sup id=\"rdp-ebb-cite_ref-LigorComp14_21-0\" class=\"reference\"><a href=\"#cite_note-LigorComp14-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChiesaAccel17_22-0\" class=\"reference\"><a href=\"#cite_note-ChiesaAccel17-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HuSimult20_23-0\" class=\"reference\"><a href=\"#cite_note-HuSimult20-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NingAccel20_24-0\" class=\"reference\"><a href=\"#cite_note-NingAccel20-24\">[24]<\/a><\/sup> Furthermore, to avoid potentially inaccurate reporting, matrix-matched standards were used for calibration. Finally, the more traditional headspace syringe (HS syringe) and liquid injection syringe (LI syringe) approaches were compared to the more modern HS-solid-phase microextraction Arrow (HS-SPME Arrow) and DI-SPME Arrow, which have recently demonstrated enhanced robustness and improved sensitivity over traditional SPME fibers.<sup id=\"rdp-ebb-cite_ref-HerringtonHunting20_25-0\" class=\"reference\"><a href=\"#cite_note-HerringtonHunting20-25\">[25]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Experimental\">Experimental<\/span><\/h2>\n<p>The following experimental sections describe the detailed procedures utilized during the three main parts of this manuscript: \n<\/p><p>1. HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow for the determination of the preferred sample introduction approach with the use of terpenes in solution;\n2. Terpene extraction evaluation for the evaluation of an ideal terpene extraction method for cannabis flower; and\n3. Combining the information gathered from the prior two procedures for a final comparison with an existing validated LI syringe technique (i.e., validated with the California Bureau of Cannabis Control [BCC]), outlined in DI-SPME Arrow vs. LI syringe.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Materials_and_reagents\">Materials and reagents<\/span><\/h3>\n<p>Hop pellets were obtained from the Beverage Factory (San Diego, CA, United States) and stored at \u221210 \u00b0C for one hour. Dried cannabis material was obtained from Cream of the Crop (San Diego, CA, United States) and stored at \u221210 \u00b0C for one hour. Cannabis Terpene Standards #1 and #2 (cat# 34095 and 34096) were purchased from Restek Corporation (Bellefonte, PA, United States). Napthalene-d8 was purchased from Restek Corporation (Bellefonte, PA, United States). Isopropanol (IPA) was purchased from Filtrous (LCMS Grade) and 1.1 mm, 120 \u00b5m DVB\/PDMS SPME Arrows were purchased from Restek Corporation (Bellefonte, PA, United States).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"HS_syringe_vs._HS-SPME_Arrow_vs._DI-SPME_Arrow\">HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow<\/span><\/h3>\n<p>The following experiments were conducted to evaluate the differences between HS syringe, HS-SPME Arrow, and DI-SPME Arrow as sample introduction techniques. Each technique was evaluated using Cannabis Terpene Standard #1 and #2 (cat# 34095 and 34096) from Restek Corporation (Bellefonte, PA, United States). Samples were evaluated using the same GC-MS conditions shown in Supplementary Table S1. For the HS syringe and HS-SPME Arrow samples, a standard stock solution was made by diluting both standards into one solution for a final concentration of 5 \u03bcg\/mL in IPA. Samples were prepared by adding 1.5 g of NaCl to a 20 mL HS vial, followed by 1 mL of 5 \u03bcg\/mL stock solution and 4 mL of water for a final concentration of 1 \u03bcg\/mL (see sampling conditions in Supplementary Tables S2, S3). Previous work (results not shown) demonstrated that HS-SPME Arrow analyte responses were higher and more reproducible when using an incubation temperature of 40 \u00b0C or less, hence having lower incubation temperature compared to the HS syringe method. The DI-SPME Arrow samples were prepared by diluting both standards into one stock solution for a final concentration of 20 \u03bcg\/mL. To a 20 mL HS vial, 1 mL of the 20 \u03bcg\/mL stock solution was added, followed by 19 mL of water for a final concentration of 1 \u03bcg\/mL (same concentration for HS syringe and HS-SPME Arrow; see sampling conditions in Supplementary Table S3). Each technique was run in triplicate for the initial evaluation of sample introduction techniques.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Terpene_extraction_evaluation\">Terpene extraction evaluation<\/span><\/h3>\n<p>An evaluation of the terpene extraction processes was conducted to understand the advantages and limitations of certain techniques. Extractions using the Dionex Accelerated Solvent Extractor (ASE 350) were compared to a hand-solvent extraction for terpene analysis. Three chemical varieties (chemovars) were used to evaluate both extraction techniques. Cannabis flower was frozen at \u221210 \u00b0C for one hour, then homogenized on a sheet pan with a rolling pin. For ASE 350 extractions, 0.5 g of homogenized cannabis flower was weighed and added to a 10 mL stainless steel ASE 350 cell, and the remaining cell volume was lightly packed with diatomaceous earth. The cell was then extracted using the parameters in Table 1. The extract was then diluted to 12 mL in a graduated cylinder due to both convenience and the fact that this approach achieved the desired data quality objectives of this study (e.g., method precision RSDs <15%). However, future researchers may consider the use of volumetric flasks to achieve better precision. One mL of the cannabis flower extract was added to a 2.5 mL autosampler vial and then analyzed. For hand extractions, 0.5 g of homogenized cannabis flower was weighed and added to a 50 mL centrifuge tube, followed by 12 mL of IPA. Samples were vortexed for three minutes and sonicated at 40 \u00b0C for five minutes. Samples were then centrifuged in a Sorvall RT7 Plus centrifuge for three minutes. One mL of the supernatant was added to a 2.5 mL autosampler vial and then analyzed. ASE 350 and hand extractions were analyzed via GC-FID.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig1_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"fd961c905c8786b73a04f12170b6727e\"><img alt=\"Fig1 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/6\/6e\/Fig1_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 1.<\/b> Accelerated solvent extractor (ASE) parameters for extracting terpenes from hops pellets and cannabis flower<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"DI-SPME_Arrow_vs._LI_syringe\">DI-SPME Arrow vs. LI syringe<\/span><\/h3>\n<p>Results from the experiments outlined in the prior two steps indicated DI-SPME Arrow was the preferred sample introduction approach, and ASE was the ideal terpene extraction technique for cannabis. This information was then utilized for a comparison to an existing validated LI syringe method. However, the experiments conducted in HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow were carried out in Pennsylvania, and the use of cannabis flower necessitated a fully licensed laboratory, which was located in California. The DI-SPME Arrow parameters, outlined in Table 2, had been further optimized for terpenes analysis in cannabis and used in the California laboratory; however, they are only slightly different from the initial DI-SPME Arrow parameters, as outlined in Supplementary Table S3. In addition, a gas chromatography\u2013<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Tandem_mass_spectrometry\" title=\"Tandem mass spectrometry\" class=\"wiki-link\" data-key=\"573d237c370deba1a7ef999284ea1cf8\">tandem mass spectrometry<\/a> (GC-MS\/MS) method was used in single quad MS mode in the California laboratory, since single quad MS is more common for this analysis (see parameters in Table 3). Furthermore, a <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Selected_ion_monitoring\" title=\"Selected ion monitoring\" class=\"wiki-link\" data-key=\"da3ef67e0e61dd34593eda73f4b2fc85\">selected ion monitoring<\/a> (SIM) method (Supplementary Table S4) was utilized to help eliminate background noise and provide better sensitivity. LI syringe was only evaluated in California using the parameters listed in Table 3.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Tab2_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"293542f9bcf10f725bbdb6c3ee9428af\"><img alt=\"Tab2 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/8\/88\/Tab2_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 2.<\/b> Optimized and final DI-SPME Arrow parameters<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Tab3_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"e8b985286dbf97fbe2b449a1b1b3bc96\"><img alt=\"Tab3 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/thumb\/7\/76\/Tab3_Myers_FrontChem2021_9.jpg\/1000px-Tab3_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 3.<\/b> Analytical parameters for evaluating terpenes in cannabis with DI-SPME Arrow and LI-Syringe.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"Hops_pellets_and_cannabis_flower_preparation\">Hops pellets and cannabis flower preparation<\/span><\/h4>\n<p>Hops pellets were utilized as a terpene-free surrogate to matrix match cannabis flower for the following DI-SPME Arrow vs. LI syringe data: calibration curves, laboratory control samples (LCS), continuing calibration verification (CCV) samples, detection limit, and analytical precision samples. Hops were crushed and homogenized on a sheet pan with a rolling pin. The crushed hops were then cleaned with a proprietary <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Solvent\" title=\"Solvent\" class=\"wiki-link\" data-key=\"978fd0f359891b1cb3927105fa01e23f\">solvent<\/a> cleaning process to eliminate the presence of terpenes. Following solvent cleaning, the hops were dried in an oven. For the DI-SPME Arrow and LI syringe method precision experiments, cannabis shake (small pieces of cannabis flower that break off of larger buds) was homogenized and utilized. For the cannabis chemovar experiments, the flower was crushed and homogenized on a sheet pan using a rolling pin.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Accelerated_solvent_extractor\">Accelerated solvent extractor<\/span><\/h4>\n<p>The following DI-SPME Arrow vs. LI syringe experiments were conducted utilizing hops pellets and cannabis flowers, which were extracted using an ASE 350 with the parameters previously shown in Table 1. For all of the following DI-SPME Arrow and LI syringe experiments, either 0.5 g of cleaned hops or 0.5 g of cannabis flower was weighed out and placed into a 10 mL ASE 350 stainless steel extraction cell. Diatomaceous earth was then slowly added and lightly packed to fill the remaining volume in the cell. Samples were then extracted using IPA. Other work has shown that extracting with IPA can lead to poor peak shape for the terpenes of interest.<sup id=\"rdp-ebb-cite_ref-KrillAHigh20_26-0\" class=\"reference\"><a href=\"#cite_note-KrillAHigh20-26\">[26]<\/a><\/sup> However, IPA gave desirable peak shape for this study and was used because of its cost, convenience, and toxicity relative to other solvents demonstrated for cannabis extractions.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"After_ASE_processing\">After ASE processing<\/span><\/h4>\n<p>After ASE extraction, all extracts, which were typically between 10 and 11 mL, were brought to a final volume of 12 mL in order to consistently evaluate extracts of the same volume. Using a 3 mL Luer lock syringe with 0.22 \u00b5m filter, 3 mL of extract was filtered. For DI-SPME Arrow experiments, 1 mL of the filtered extract was added to 19 mL of LCMS grade water (i.e., 20 mL final volume) in a 20 mL headspace vial. In addition, 20 \u00b5L of 100 \u03bcg\/mL internal standard (ISTD) solution was added. Subsequently, the headspace vial was capped and spun for 10 seconds. For LI syringe experiments, 500 \u00b5L of the filtered extract was added to a 2 mL autosampler vial. In addition, 5 \u00b5L of the 10 \u03bcg\/mL ISTD solution was added. Subsequently, the autosampler vial was capped and spun for 10 seconds.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Terpenes_standards_and_internal_standards\">Terpenes standards and internal standards<\/span><\/h4>\n<p>Differences in linear range between DI-SPME Arrow and LI syringe necessitated the use of the different intermediate and ISTD solutions. Intermediate concentrations of 1000 \u03bcg\/mL and a 10 \u03bcg\/mL were prepared from the 2,500 \u03bcg\/mL Terpene Standards 1 and 2. To prepare the 1000 \u03bcg\/mL intermediate, 400 \u00b5L of each terpene standard (i.e., 800 \u00b5L total) was added to 200 \u00b5L of IPA, then capped and vortexed. The 10 \u03bcg\/mL intermediate was prepared from the 1000 \u03bcg\/mL intermediate by adding 10 \u00b5L of the 1000 \u03bcg\/mL intermediate to 990 \u00b5L of IPA, then capped and vortexed. A solution of naphthalene-d8 ISTD was made at 100 \u03bcg\/mL for DI-SPME Arrow experiments and 10 \u03bcg\/mL for LI syringe experiments.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"DI-SPME_Arrow_calibration\">DI-SPME Arrow calibration<\/span><\/h4>\n<p>For the highest DI-SPME Arrow calibration point (level 7), 153.6 \u00b5L of 1000 \u03bcg\/mL terpene solution was added to hops. Once extracted, the extract was brought to 12 mL with IPA and filtered, thereby reducing calibration level 7\u201312.8 \u03bcg\/mL. Intermediate serial dilutions (Supplementary Table S5) were carried out on calibration level 7 to make the other six calibration points. For example, 1500 \u00b5L of calibration level 7 was added to 1500 \u00b5L of IPA to make calibration level 6. This process was then repeated for the other calibration points. However, the final calibration solutions required a secondary dilution into 20 mL headspace vials (Supplementary Table S6). For example, 1 mL of the calibration level 6 (i.e., 6.4 \u03bcg\/mL) was added to 19 mL of water (i.e., 20 mL total volume) for a final concentration of 0.32 \u03bcg\/mL and then spiked with ISTD. For a DI-SPME Arrow CCV equivalent to calibration level 3, calibration 7 filtered extract was diluted with IPA, spiked with ISTD, capped, and vortexed for 10 seconds.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"LI_syringe_calibration\">LI syringe calibration<\/span><\/h4>\n<p>For the highest LI syringe calibration point (level 10), 61.4 \u00b5L of the 1000 \u03bcg\/mL terpene solution was added to the hops. Once extracted, the extract was brought to 12 mL with IPA and filtered, thereby reducing calibration level 10 to 5.12 \u03bcg\/mL. Serial dilutions (Supplementary Table S7) were carried out on calibration level 10 to make the other nine calibration points. For example, 500 \u00b5L of calibration level 10 was added to 500 \u00b5L of IPA to make calibration level 9. This process was then repeated for the other calibration points. Finally, 5 \u00b5L of the 10 \u03bcg\/mL ISTD solution was added to each calibration vial at levels 2\u201310, and 10 \u00b5L of the 10 \u03bcg\/mL ISTD solution was added to level 1 given the difference in final volume. After being spiked with ISTD, the vial was capped and spun for 10 seconds. See Supplementary Table S7 for the LI syringe calibration curve.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Method_validation_and_chemovar_experiments\">Method validation and chemovar experiments<\/span><\/h4>\n<p>This section addresses the following method validations: method detection limit (MDL)\/limit of quantitation (LOQ), analytical precision, method precision, and % recovery. The DI-SPME Arrow and LI syringe MDLs\/LOQs were determined from seven replicate low-level calibration points. In addition, LCSs were run to determine the analytical precision and % recovery of both methods. For a DI-SPME Arrow LCS, 76.8 \u00b5L of the 1000 \u03bcg\/mL intermediate terpene solution was added to hops (equivalent to calibration level 6). For an LI syringe LCS, 384 \u00b5L of 10 \u03bcg\/mL intermediate terpene solution was added to the hops (equivalent to calibration level 6). It is important to note that the LCS represents a separate hops spike and extraction. Furthermore, the DI-SPME Arrow and LI syringe method precisions were determined from seven different aliquots of cannabis shake. Finally, three different chemovars of cannabis flower were evaluated for terpenes with DI-SPME Arrow and LI syringe.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"HS_syringe_vs._HS-SPME_Arrow_vs._DI-SPME_Arrow_2\">HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow<\/span><\/h3>\n<p>Initial work compared three different types of sample preparation\/introduction techniques for terpene analysis via GC-MS. Techniques were evaluated based on relative compound response using only reference terpene standards. First, the more traditional approach using a HS syringe was compared to HS-SPME Arrow (120 \u00b5m DVB\/PDMS). As shown in Figure 1, 13 of 23 terpenes were identified using the HS syringe approach. However, this approach was unable to effectively pick up the later eluting and less volatile terpenes, which fall into the sesquiterpene category. When samples were analyzed via HS-SPME Arrow, 23 of 23 terpenes were able to be identified.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig1Act_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"094902c6215a607ef21cba7a8ec0b30c\"><img alt=\"Fig1Act Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/8\/88\/Fig1Act_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Analytical parameters for evaluating terpenes in cannabis with DI-SPME Arrow and LI-Syringe<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>When comparing responses for the 13 terpenes that were able to be identified in both approaches, HS-SPME Arrow had much greater responses than the HS syringe approach. For the terpenes found in both HS techniques, the responses on the SPME Arrow were >10\u00d7 that of the HS syringe. Both samples were prepared identically and analyzed with the suggested parameters for each technique. When first looking at the HS syringe results, it was unclear if the less volatile sesquiterpenes were partitioning into the HS of the 20 mL vial. However, after analyzing the results for the HS-SPME Arrow and detecting the less volatile compounds, it was confirmed that these compounds are partitioning into the HS of the vial. It is not clear as to where the terpenes were lost (i.e., not transferred efficiently) in the HS syringe process, and it was outside of the scope of this study to determine the root cause. Because the HS-SPME Arrow method was able to identify all of the terpenes in the samples, this approach was chosen to move forward in the study. However, it was desired to see how HS-SPME Arrow compared to DI-SPME Arrow.\n<\/p><p>HS-SPME Arrow samples and DI-SPME Arrow samples were prepared according to their respective approach, but they were analyzed under the same instrument conditions. Both techniques were able to identify all terpenes within the reference standard samples. However, as shown in Figure 2, terpene samples analyzed via DI-SPME Arrow showed improved responses over HS-SPME Arrow, especially for the higher molecular weight terpenes and also proved to be more reproducible (i.e., provide better precision). Responses for the DI-SPME Arrow averaged 6\u00d7 greater than that of the HS-SPME Arrow. %RSDs for the HS-SPME Arrow were as high as 76%, while all DI-SPME Arrow %RSDs were \u226415%. Potential limitations of DI-SPME Arrow include shortened fiber lifetime and\/or increased matrix exposure; however, given the improved responses and reproducibility, it was selected as the technique to move forward for further method validation and compared against a traditional liquid injection (LI) syringe method.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig2_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"36fd2780de816a5944c061cd63777d09\"><img alt=\"Fig2 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/f\/f4\/Fig2_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> HS-SPME Arrow vs DI-SPME Arrow for terpenes<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Hand_shakeout_vs._accelerated_solvent_extractor\">Hand shakeout vs. accelerated solvent extractor<\/span><\/h3>\n<p>Several terpene extraction approaches were considered for the current study. The full evaporative technique (FET), which is popular within the cannabis testing industry, was not evaluated in the current study as this technique\u2019s foundation is HS syringe. Additionally, the results discussed in HS syringe vs. HS-SPME Arrow vs. DI-SPME Arrow demonstrated that HS syringe did not perform as well as HS-SPME Arrow, which was also inferior to DI-SPME Arrow for the analysis of terpenes. Other industries already capitalize on the benefits of ASE 350<sup id=\"rdp-ebb-cite_ref-LigorComp14_21-1\" class=\"reference\"><a href=\"#cite_note-LigorComp14-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChiesaAccel17_22-1\" class=\"reference\"><a href=\"#cite_note-ChiesaAccel17-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HuSimult20_23-1\" class=\"reference\"><a href=\"#cite_note-HuSimult20-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NingAccel20_24-1\" class=\"reference\"><a href=\"#cite_note-NingAccel20-24\">[24]<\/a><\/sup> Therefore, a simple hand shakeout solvent extraction method was compared to an ASE 350 extraction method to evaluate the performance of each technique for extracting terpenes from cannabis flower. Three different cannabis chemovars were extracted using both techniques, and the average of their FID responses were determined (Table 4). Both techniques extracted the same 13 terpenes from the cannabis flower. On average, the hand shakeout responses were better than the ASE 350 responses for 11 of the 13 terpenes detected. Given the small sample size, a nonparametric Kruskal-Wallis test was completed to compare the averages and determine if there was a statistical difference between the hand shakeout and ASE 350 approaches. With the exception of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Camphene\" title=\"Camphene\" class=\"wiki-link\" data-key=\"0876f83286b041512522980a32ec2969\">camphene<\/a> and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Linalool\" title=\"Linalool\" class=\"wiki-link\" data-key=\"065d30b67ffb79c4152ca0e04d58d40d\">linalool<\/a> (<i>p<\/i> = 0.050), the Kruskal\u2013Wallis tests indicate a general trend of no statistically significant difference between the hand shakeout and ASE 350 extraction techniques for the 13 detected terpenes.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Tab4_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"c1ec5cb97dbe8f679ea6f15a40246c54\"><img alt=\"Tab4 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/d\/d6\/Tab4_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 4.<\/b> Hand shakeout vs. accelerated solvent extraction (ASE 350) for extraction of terpenes from cannabis flower<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In addition, several factors were considered when selecting the extraction method. The hand shakeout required more use of nonreusable consumables (e.g., centrifuge tubes), making it a \u201cgreener\u201d approach. In addition, the cannabis flower hand shakeout technique could potentially differ depending on the lab technician completing the manually intensive hand shakeout extraction. On the contrary, the ASE 350 cells were reused by cleaning after extraction. Furthermore, the extraction consistency that the ASE 350 offers was not user dependent. The lack of statistical significance between the two extraction techniques, coupled with less consumable needs and user variability, leads to utilizing the ASE 350 for all of the following DI-SPME Arrow vs. LI syringe experiments. It is important to note that this was the first study to utilize an ASE 350 for the extraction of terpenes from cannabis. Despite this novel development for the field of cannabis, future studies should consider the further development of the current ASE 350 parameters to optimize extraction efficiency by changing solvent extraction ratios, extraction temperature, etc. In addition, this technique lends itself very well to joint application potential with <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_(drug)#Varieties_and_strains\" title=\"Cannabis (drug)\" class=\"wiki-link\" data-key=\"e89f27be6689b63296ef81a36148b763\">potency<\/a> and\/or <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Pesticide\" title=\"Pesticide\" class=\"wiki-link\" data-key=\"073c24701223ff2ad1d3e2cc0058b2de\">pesticides<\/a>, therefore making it a very desirable technique.<sup id=\"rdp-ebb-cite_ref-RestekFast20_27-0\" class=\"reference\"><a href=\"#cite_note-RestekFast20-27\">[27]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Hops_for_clean_surrogate_matrix\">Hops for clean surrogate matrix<\/span><\/h3>\n<p>An initial comparison of identical calibration curves prepared in 100% IPA and matrix demonstrated matrix effects (MEs).<sup id=\"rdp-ebb-cite_ref-RaposoChall21_20-1\" class=\"reference\"><a href=\"#cite_note-RaposoChall21-20\">[20]<\/a><\/sup> Of the 23 terpenes evaluated, 17 had positive MEs (seven average), as defined by Chambers <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ChambersSystem07_28-0\" class=\"reference\"><a href=\"#cite_note-ChambersSystem07-28\">[28]<\/a><\/sup> This observation suggested there was a \u201csoft\u201d signal enhancement for these compounds, which may be the result of matrix-induced <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Chromatography\" title=\"Chromatography\" class=\"wiki-link\" data-key=\"1b40e146652470be00cebaf949c68b24\">chromatographic<\/a> response.<sup id=\"rdp-ebb-cite_ref-Hajslov.C3.A1Matrix03_29-0\" class=\"reference\"><a href=\"#cite_note-Hajslov.C3.A1Matrix03-29\">[29]<\/a><\/sup> The remaining six terpenes (linalool, isopulegol, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Geraniol\" title=\"Geraniol\" class=\"wiki-link\" data-key=\"80bd9524b34df724b9c0d400ce2f3976\">geraniol<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Caryophyllene\" title=\"Caryophyllene\" class=\"wiki-link\" data-key=\"2b57176900e3cbad5be1ea724ba37cfd\">\u03b2-caryophyllene<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Humulene\" title=\"Humulene\" class=\"wiki-link\" data-key=\"9c05eef99c1446cac40b10d57a75ccff\">\u03b1-humulene<\/a>, and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Nerolidol\" title=\"Nerolidol\" class=\"wiki-link\" data-key=\"7ef6af4c2e9d806372ecc031ce042896\">trans-nerolidol<\/a>) had negative MEs (-18 average), which suggested a \u201csoft\u201d signal suppression. The signal suppression for the aforementioned alcohols is contradictory to the theory of matrix-induced chromatographic response and represents a testament to the complexities of matrix effects. Regardless, these results indicated that cannabis flower MEs were present and therefore a matrix match calibration approach was deemed ideal. However, it was outside the scope of the current manuscript to fully dissect all of the current ME phenomenon associated with cannabis, especially considering the wide array of cannabis matrices. Future researchers are encouraged to expand upon the current start to understanding cannabis MEs.\n<\/p><p>To date, the cannabis industry has not been utilizing surrogate matrices, despite the fact that studies have demonstrated matrix effects could lead to inaccurate reporting.<sup id=\"rdp-ebb-cite_ref-RaposoChall21_20-2\" class=\"reference\"><a href=\"#cite_note-RaposoChall21-20\">[20]<\/a><\/sup> Due to the numerous types of matrices cannabis testing laboratories analyze, laboratories are forced to become creative when doing matrix matching for their calibrations. Matrix matching for terpenes in cannabis flower represents a particularly tough issue, because similar plant species to cannabis also contain terpenes. In this study, a novel method of cleaning hops was utilized to provide a terpene-free surrogate for matrix matched calibrations. Matrix blanks were run to demonstrate the cleaned hops were free of terpenes and did not contribute to compound responses for the terpenes of interest (Figure 3). This was the first study to utilize cleaned hops as a clean surrogate for cannabis flower, and the following method validation results demonstrate not only that this was a viable technique, but also it produced a desirable outcome (i.e., excellent method performance).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig3_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"a802a69a7ce3ec47a440007340c23ece\"><img alt=\"Fig3 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/2\/2b\/Fig3_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> Cleaned hops blank (LI syringe) demonstrating terpene-free surrogate matrix<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Method_performance_.28DI-SPME_Arrow_vs._LI_syringe.29\">Method performance (DI-SPME Arrow vs. LI syringe)<\/span><\/h3>\n<p>Method validation was done in accordance with the California Bureau of Cannabis Control guidelines and regulations.<sup id=\"rdp-ebb-cite_ref-CBCC16-42_30-0\" class=\"reference\"><a href=\"#cite_note-CBCC16-42-30\">[30]<\/a><\/sup> Terpenes were analyzed via GC-MS\/MS using SIM in single quad MS mode. Single quad MS mode was used to be more representative of most cannabis laboratories, and SIM was used to minimize matrix interferences. All terpenes of interest were resolved using the Rxi-624Sil MS, 30 m \u00d7 0.25 mm \u00d7 1.4 \u00b5m (cat# 13868), as shown in Figure 4.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig4_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"2141acc5e68b6b685ec8ded26b83f8ba\"><img alt=\"Fig4 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/f\/fa\/Fig4_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4.<\/b> Full separation of 23 terpenes with GC-MS (LI syringe)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"DI-SPME_Arrow_terpene_validation\">DI-SPME Arrow terpene validation<\/span><\/h3>\n<p>Terpene method validation was completed for DI-SPME Arrow to evaluate performance and possible implementation into cannabis testing laboratories. Method performance can be seen in Table 5. As shown in Table 5, an average <i>r<\/i><sup>2<\/sup> value of 0.991 was achieved for the terpenes of interest in an average working range of 0.08\u20130.64 \u03bcg\/mL. <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Pinene\" title=\"Pinene\" class=\"wiki-link\" data-key=\"6de2eb2cc03bfd4404a442d2f0e4076b\">\u03b1-pinene<\/a> had the lowest <i>r<\/i><sup>2<\/sup> value at 0.965. While most compounds had a working range of 0.08\u20130.64 \u03bcg\/mL for calibration, a few compounds (e.g., <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Bisabolol\" title=\"Bisabolol\" class=\"wiki-link\" data-key=\"89705d0817b73e1c72693af6678ce1cb\">\u03b1-bisabolol<\/a>) were able to have expanded ranges from 0.04\u20130.64 \u03bcg\/mL and 0.02\u20130.64 \u03bcg\/mL. However, two compounds (\u03b2-caryophyllene and \u03b1-humulene) had shortened ranges from 0.08 to 0.32 \u03bcg\/mL. The differences in calibration working ranges can be attributed to each individual compound\u2019s ability to ad\/absorb to the phase on the SPME Arrow and desorb, which can lead to the compound saturation. When compound saturation occurs within the calibration, the calibration curve will plateau, resulting in poor <i>r<\/i><sup>2<\/sup> values. Previous work looked at higher calibration ranges (>1 \u03bcg\/mL); however, because the Arrow phase reached its saturation point for this compound class, the calibration range needed to be lowered to a more appropriate linear range. This phenomenon can also be combated by adjusting the parameters (e.g., extraction time, desorption time, etc.), but changing these parameters could then lead to poor results for additional compounds.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Tab5_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"0881a1dacbb5ebc52c76b2551b15026e\"><img alt=\"Tab5 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/5\/59\/Tab5_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 5.<\/b> DI-SPME Arrow\u2013GC-MS method validation with ASE extracted cleaned hops matrix match for cannabis<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>An average LOQ of 0.050 \u03bcg\/mL was achieved for the DI-SPME Arrow method. The LOQ was three times the MDL, which was determined off of seven (<i>n<\/i> = 7) replicate injections at 0.08 \u03bcg\/mL. MDLs were calculated as the standard deviation of the seven replicate measurements multiplied by 3.14 (i.e., the Student\u2019s <i>t<\/i>-value for 99% confidence for seven values). Analytical precision was evaluated using reference standards. Seven replicate injections were made and %RSDs fell below 10% (average = 6.77%) for all terpenes of interest, proving that DI-SPME Arrow has the ability to be a robust analytical technique. In addition to analytical precision, overall method precision (i.e., ASE 350 and DI-SPME Arrow GC-MS) was evaluated by analyzing cannabis shake. Seven shake extractions were made and one sample from each extraction was analyzed to evaluate method precision. As seen in Table 5, there is a variance in %RSDs between the different terpenes of interest, and not all terpenes were detected in the shake. Overall, an average %RSD of 14.89% was achieved for the 17 terpenes detected in the cannabis shake. With 14 of the 17 detected terpenes having acceptable %RSDs (i.e., <30%), the remaining three terpenes (e.g., \u03b2-caryophyllene) having borderline unacceptable %RSDs. The % recoveries ranged from 67.2 to 262%, with an average of 95.5%. Three compounds (<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Terpinene\" title=\"Terpinene\" class=\"wiki-link\" data-key=\"3ec3ee781b6e0b7466b1777e5482c813\">\u03b1-terpinene<\/a>, terpinolene, and \u03b2-caryophyllene) had recoveries outside the acceptable \u00b130% window and were 262, 172, and 67.2%, respectively.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Liquid_injection_syringe_terpene_validation\">Liquid injection syringe terpene validation<\/span><\/h3>\n<p>Method performance was evaluated for a LI syringe method and can be seen in Table 6. Over an average calibration working range of 0.04\u20135.12 \u03bcg\/mL, an average <i>r<\/i><sup>2<\/sup> of 0.993 was achieved for the 23 terpenes of interest. \u03b2-caryophyllene and \u03b1-bisabolol had the smallest calibration range spanning from 0.17 to 5.12 \u03bcg\/mL, but maintained good <i>r<\/i><sup>2<\/sup> values of 0.992 and 0.990, respectively. Twenty-one of the 23 terpenes of interest had an <i>r<\/i><sup>2<\/sup> value of \u22650.990, while the additional two compounds (geraniol and trans-nerolidol) had an <i>r<\/i><sup>2<\/sup> value of 0.988. It is important to note that the LI syringe technique was able to achieve a higher range for the terpenes.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Tab6_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"749e74c6265c580d828b2586a5308331\"><img alt=\"Tab6 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/9\/91\/Tab6_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 6.<\/b> LI syringe\u2013GC-MS method validation with ASE extracted cleaned hops matrix for cannabis<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>An average LOQ of 0.047 \u03bcg\/mL was achieved for the LI syringe method, which is nearly identical to the average LOQ achieved for the DI-SPME Arrow method. LOQs for the LI syringe method were calculated identically to what was completed to those for the DI-SPME Arrow method. It should be noted that while the LI syringe method had lower LOQs for the more volatile terpenes of interest, the DI-SPME Arrow method had lower LOQs for the majority of the less volatile terpenes of interest. Analytical precision had an average %RSD of 1.62%, with all compounds below 5%. Improvements were made when looking at method precision via LI syringe vs. DI-SPME Arrow. All terpenes of interest had %RSDs of less than 10%, with the exception of cis-nerolidol (<15%) for method precision. Average %RSD for LI syringe was 4.75%. The % recoveries ranged from 84.6 to 98.9%, with an average of 90.2%. All 23 terpenes of interest fell into the acceptable \u00b130% window for the LI syringe method.\n<\/p><p>The aforementioned results are comparable to a previously published study by Ibrahim <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-IbrahimAnal19_12-3\" class=\"reference\"><a href=\"#cite_note-IbrahimAnal19-12\">[12]<\/a><\/sup>, which showed <i>r<\/i><sup>2<\/sup> value of >0.99, %RSDs <15%, recoveries ranging from 67 to 105.7%, MDLs of 0.25 \u03bcg\/mL, and LOQs of 0.75 \u03bcg\/mL. The current study was able to build off of the foundation from Ibrahim <i>et al.<\/i> and achieved similar <i>r<\/i><sup>2<\/sup> values, %RSDs, and recoveries. However, this study excelled with lower working calibration range and an order of magnitude lower LOQs (0.047 \u03bcg\/mL average). In addition, this study employed the use of accelerated solvent extraction and matrix matched calibrations.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Terpene_chemovar_evaluation_with_DI-SPME_Arrow_and_LI_syringe\">Terpene chemovar evaluation with DI-SPME Arrow and LI syringe<\/span><\/h3>\n<p>After method validation, the chemical variety (chemovar) of three different cannabis chemovars was evaluated via ASE and both DI-SPME Arrow and LI syringe techniques. Results shown in Table 7 represent the concentration in the sample for Mint Chocolate Chip (i.e., corrected for ASE extraction and dilution). It is important to note that initially three different chemovars (<i>n<\/i> = 9 total) were evaluated by both DI-SPME Arrow and LI syringe; however, due to SPME Arrow saturation on the first two chemovars, the Mint Chocolate Chip chemovar underwent a secondary 10\u00d7 dilution of the cannabis extract. From the data collected, several points should be discussed when comparing the two methods. For the DI-SPME Arrow method, 17 of 23 terpenes were identified and 22 of 23 were identified via LI syringe. It should be noted that one compound (<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=P-Cymene\" title=\"P-Cymene\" class=\"wiki-link\" data-key=\"06005b4dd1e60b101c46f20f2055e2c0\">p-cymene<\/a>) was not detected (ND) by either technique and should be considered as not present in this chemovar. The DI-SPME Arrow method had several compounds falling outside of the working ranges and several compounds that were ND. It is hypothesized that perhaps the SPME Arrow used for this experiment was reaching the end of its viable lifetime and therefore was not as efficiently adsorbing and\/or releasing the terpenes; however, this was not determined. Regardless, the LI syringe appeared to provide better results when analyzing a true sample. Nearly all compounds fell within the LI syringe calibration working ranges and saturation was not an issue.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Tab7_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"1e921f113fb0c4df235e4e122dc934ec\"><img alt=\"Tab7 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/1\/15\/Tab7_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 7.<\/b> Mint Chocolate Chip terpene content determined by DI-SPME Arrow and LI syringe<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Despite the aforementioned discrepancies, the average concentration for DI-SPME Arrow was 18.3 \u03bcg\/g, while the LI syringe average concentration was 16.7 \u03bcg\/g. Comparing the average concentrations specifically, both techniques showed similar concentrations for half of the compounds and divergent concentrations for the other half. More importantly, when comparing the overall terpene profiles of each injection technique, the profiles showed similar results (Figure 5). In particular, both DI-SPME Arrow and LI syringe shared the same top five most abundant compounds, which include \u03b1-pinene, \u03b2-pinene, D-limonene, linalool, and \u03b2-caryophyllene. Granted, \u03b1-pinene\u2019s response was almost 2\u00d7 greater for DI-SPME Arrow, while \u03b2-caryophyllene\u2019s response was over 2.5\u00d7 greater for LI syringe. Alternatively, the percentages for D-limonene and linalool fell within 3% of one another. Note that the other 18 terpenes were summed in the \u201cother\u201d category. The LI syringe had more positive hits for lower concentration terpenes, as its calibration range was lower than the DI-SPME Arrow, hence having the 29% \u201cother\u201d for LI syringe vs. the 15% \u201cother\u201d for DI-SPME Arrow.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig5_Myers_FrontChem2021_9.jpg\" class=\"image wiki-link\" data-key=\"258311b5ccef2ab971299d8c95144041\"><img alt=\"Fig5 Myers FrontChem2021 9.jpg\" src=\"https:\/\/www.cannaqa.wiki\/images\/thumb\/5\/57\/Fig5_Myers_FrontChem2021_9.jpg\/400px-Fig5_Myers_FrontChem2021_9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> Mint Chocolate Chip terpene profiles determined by DI-SPME Arrow and LI syringe<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Conclusions.2C_future_work.2C_and_recommendations\">Conclusions, future work, and recommendations<\/span><\/h2>\n<p>The current study demonstrated that DI-SPME Arrow performed better than HS-SPME Arrow; however, both of these approaches outperformed HS syringe for the extraction and analyses of terpenes. A novel method for cleaning hops was developed to provide terpene-free hops utilized as a surrogate for matrix matched calibrations. For the first time, both hops and cannabis flower were extracted with ASE and then utilized for method validation.\n<\/p><p>Validated methodologies for analyzing terpenes in cannabis flower were developed for both DI-SPME Arrow and liquid injection syringe with GC-MS\/MS in single quad mode and selected ion monitoring. Both methods proved to be viable options for the analysis of terpenes in cannabis flower. When comparing the average LOQs of the terpenes of interest, both techniques were near identical. However, results suggest that LI syringe would be the preferred approach for this analysis based on several observations. The LI syringe showed an expanded working calibration range compared to the DI-SPME Arrow, which can be attributed to phase saturation of the SPME Arrow. In addition, better analytical and method precision was achieved by LI syringe. Furthermore, LI syringe appeared to provide a more complete chemovar profile of cannabis flower and at higher concentrations. Lastly, SPME Arrows needed routine replacement due to phase swelling and\/or lifetime, a fate which was not shared by LI syringe.\n<\/p><p>While results indicate that LI syringe is the preferred technique, other factors should be considered for future work. When running LI syringe, more GC maintenance may be needed when compared to the popular HS methods in the cannabis testing industry. This includes inlet consumable changes, analytical column trimming, and MS maintenance, which will come as a result of the matrix being injected into the system. One area not explored in the current work that should be evaluated in future work is instrument uptime for DI-SPME Arrow vs. LI syringe. It is hypothesized that less matrix may be exposed to the GC when running the DI-SPME Arrow method, which in turn may lead to longer instrument uptime and less time and money spent on maintenance.\n<\/p><p>It is recommended that the scientific cannabis community reconsider utilizing HS-Syringe for the analysis of terpenes in cannabis products, as the current results suggest it is inferior to all of the approaches discussed, especially LI syringe. Therefore, it is also recommend that the full evaporative technique, which is an HS syringe-based approach, also be reassessed in the future. Furthermore, the FET approach is not amenable to splitting samples for other cannabis test methods (e.g., potency), like the ASE method outlined in this study. In addition, it is recommended that future cannabis work continue to evaluate the use of terpene-free hops for surrogate matrix matching of flower. Furthermore, it is recommended that additional cannabis research further the development of ASE methods for the extraction and analysis of terpenes. Under the appropriate conditions, the ASE has the potential to improve laboratory workflows by using one extraction and splitting that extract between multiple analyses in addition to terpene profiling (e.g., potency, pesticides, and mycotoxins). Lastly, it is recommended that this work should be expanded to additional matrices that cannabis testing laboratories frequently analyze (e.g., shatters and waxes), and expand upon the cannabis ME knowledge, which has only begun here. These additional matrices bring new challenges and will need to be addressed to improve the science of cannabis testing.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_material\">Supplementary material<\/span><\/h2>\n<ul><li> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/ndownloader.figstatic.com\/files\/27391250\">Datasheet 1<\/a> (PDF)<\/li>\n<li> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/ndownloader.figstatic.com\/files\/27391253\">Datasheet 2<\/a> (PDF)<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>CM started this study by investigating different approaches for analyzing terpenes. CM and JH outlined the research plan, analyzed data, and wrote the manuscript. PH facilitated the research by providing a licensed cannabis testing laboratory to continue the research. KA conducted the experiments outlined by CM and JH at the laboratory owned by PH.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_availability_statement\">Data availability statement<\/span><\/h3>\n<p>The original contributions presented in the study are included in the article\/Supplementary Material; further inquiries can be directed to the corresponding author.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-Stelton-HoltmeierChart20-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Stelton-HoltmeierChart20_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Stelton-Holtmeier, J. (25 August 2020). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/mjbizdaily.com\/chart-nationwide-sales-of-adult-use-cannabis-further-eclipse-those-of-medical-marijuana\/\">\"Chart: Nationwide sales of adult-use cannabis further eclipse those of medical marijuana\"<\/a>. <i>Marijuana Business Daily<\/i><span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/mjbizdaily.com\/chart-nationwide-sales-of-adult-use-cannabis-further-eclipse-those-of-medical-marijuana\/\">https:\/\/mjbizdaily.com\/chart-nationwide-sales-of-adult-use-cannabis-further-eclipse-those-of-medical-marijuana\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 31 August 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chart%3A+Nationwide+sales+of+adult-use+cannabis+further+eclipse+those+of+medical+marijuana&rft.atitle=Marijuana+Business+Daily&rft.aulast=Stelton-Holtmeier%2C+J.&rft.au=Stelton-Holtmeier%2C+J.&rft.date=25+August+2020&rft_id=https%3A%2F%2Fmjbizdaily.com%2Fchart-nationwide-sales-of-adult-use-cannabis-further-eclipse-those-of-medical-marijuana%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MDFGlobal20-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MDFGlobal20_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-MDFGlobal20_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.marketdataforecast.com\/market-reports\/cannabis-testing-market\">\"Chart: Nationwide sales of adult-use cannabis further eclipse those of medical marijuana\"<\/a>. February 2020<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.marketdataforecast.com\/market-reports\/cannabis-testing-market\">https:\/\/www.marketdataforecast.com\/market-reports\/cannabis-testing-market<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 31 August 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chart%3A+Nationwide+sales+of+adult-use+cannabis+further+eclipse+those+of+medical+marijuana&rft.atitle=&rft.date=February+2020&rft_id=https%3A%2F%2Fwww.marketdataforecast.com%2Fmarket-reports%2Fcannabis-testing-market&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NuutinenMedicinal18-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NuutinenMedicinal18_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-NuutinenMedicinal18_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nuutinen, T. (2018). \"Medicinal properties of terpenes found in <i>Cannabis sativa<\/i> and <i>Humulus lupulus<\/i>\". <i>European Journal of Medicinal Chemistry<\/i> <b>157<\/b>: 198\u2013228. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ejmech.2018.07.076\">10.1016\/j.ejmech.2018.07.076<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30096653\">30096653<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medicinal+properties+of+terpenes+found+in+%27%27Cannabis+sativa%27%27+and+%27%27Humulus+lupulus%27%27&rft.jtitle=European+Journal+of+Medicinal+Chemistry&rft.aulast=Nuutinen%2C+T.&rft.au=Nuutinen%2C+T.&rft.date=2018&rft.volume=157&rft.pages=198%E2%80%93228&rft_id=info:doi\/10.1016%2Fj.ejmech.2018.07.076&rft_id=info:pmid\/30096653&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CalviCompre18-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CalviCompre18_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-CalviCompre18_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-CalviCompre18_4-2\">4.2<\/a><\/sup> <sup><a href=\"#cite_ref-CalviCompre18_4-3\">4.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Calvi, L.; Pentimalli, D.; Panseri, S. et al. (2018). \"Comprehensive quality evaluation of medical <i>Cannabis sativa<\/i> L. inflorescence and macerated oils based on HS-SPME coupled to GC\u2013MS and LC-HRMS (q-exactive orbitrap) approach\". <i>Journal of Pharmaceutical and Biomedical Analysis<\/i> <b>150<\/b>: 208\u201319. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpba.2017.11.073\">10.1016\/j.jpba.2017.11.073<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comprehensive+quality+evaluation+of+medical+%27%27Cannabis+sativa%27%27+L.+inflorescence+and+macerated+oils+based+on+HS-SPME+coupled+to+GC%E2%80%93MS+and+LC-HRMS+%28q-exactive+orbitrap%29+approach&rft.jtitle=Journal+of+Pharmaceutical+and+Biomedical+Analysis&rft.aulast=Calvi%2C+L.%3B+Pentimalli%2C+D.%3B+Panseri%2C+S.+et+al.&rft.au=Calvi%2C+L.%3B+Pentimalli%2C+D.%3B+Panseri%2C+S.+et+al.&rft.date=2018&rft.volume=150&rft.pages=208%E2%80%9319&rft_id=info:doi\/10.1016%2Fj.jpba.2017.11.073&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RussoTaming11-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RussoTaming11_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Russo, E.B. (2011). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\">\"Taming THC: Potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\"<\/a>. <i>British Journal of Pharmacology<\/i> <b>163<\/b> (7): 1344-64. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1476-5381.2011.01238.x\">10.1111\/j.1476-5381.2011.01238.x<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\">PMC3165946<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21749363\">21749363<\/a><span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Taming+THC%3A+Potential+cannabis+synergy+and+phytocannabinoid-terpenoid+entourage+effects&rft.jtitle=British+Journal+of+Pharmacology&rft.aulast=Russo%2C+E.B.&rft.au=Russo%2C+E.B.&rft.date=2011&rft.volume=163&rft.issue=7&rft.pages=1344-64&rft_id=info:doi\/10.1111%2Fj.1476-5381.2011.01238.x&rft_id=info:pmc\/PMC3165946&rft_id=info:pmid\/21749363&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3165946&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbilleiraAnAccur10-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbilleiraAnAccur10_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abilleira, E.; de Renobales, M.; N\u00e1jera, A.I. et al. (2010). \"An accurate quantitative method for the analysis of terpenes in milk fat by headspace solid-phase microextraction coupled to gas chromatography\u2013mass spectrometry\". <i>Food Chemistry<\/i> <b>120<\/b> (4): 1162-1169. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.foodchem.2009.11.050\">10.1016\/j.foodchem.2009.11.050<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+accurate+quantitative+method+for+the+analysis+of+terpenes+in+milk+fat+by+headspace+solid-phase+microextraction+coupled+to+gas+chromatography%E2%80%93mass+spectrometry&rft.jtitle=Food+Chemistry&rft.aulast=Abilleira%2C+E.%3B+de+Renobales%2C+M.%3B+N%C3%A1jera%2C+A.I.+et+al.&rft.au=Abilleira%2C+E.%3B+de+Renobales%2C+M.%3B+N%C3%A1jera%2C+A.I.+et+al.&rft.date=2010&rft.volume=120&rft.issue=4&rft.pages=1162-1169&rft_id=info:doi\/10.1016%2Fj.foodchem.2009.11.050&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KupskaComp14-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KupskaComp14_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kupska, M.; Chmiel, T.; J\u0119drkiewicz, R. et al. (2014). \"Comprehensive two-dimensional gas chromatography for determination of the terpenes profile of blue honeysuckle berries\". <i>Food Chemistry<\/i> <b>152<\/b>: 88\u201393. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.foodchem.2013.11.129\">10.1016\/j.foodchem.2013.11.129<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24444910\">24444910<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comprehensive+two-dimensional+gas+chromatography+for+determination+of+the+terpenes+profile+of+blue+honeysuckle+berries&rft.jtitle=Food+Chemistry&rft.aulast=Kupska%2C+M.%3B+Chmiel%2C+T.%3B+J%C4%99drkiewicz%2C+R.+et+al.&rft.au=Kupska%2C+M.%3B+Chmiel%2C+T.%3B+J%C4%99drkiewicz%2C+R.+et+al.&rft.date=2014&rft.volume=152&rft.pages=88%E2%80%9393&rft_id=info:doi\/10.1016%2Fj.foodchem.2013.11.129&rft_id=info:pmid\/24444910&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CachoEval15-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CachoEval15_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cacho, J.I.; Campillo, N.; Vi\u00f1as, P. et al. (2015). \"Evaluation of three headspace sorptive extraction coatings for the determination of volatile terpenes in honey using gas chromatography-mass spectrometry\". <i>Journal of Chromatography A<\/i> <b>1399<\/b>: 18\u201324. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.chroma.2015.04.041\">10.1016\/j.chroma.2015.04.041<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25958092\">25958092<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+of+three+headspace+sorptive+extraction+coatings+for+the+determination+of+volatile+terpenes+in+honey+using+gas+chromatography-mass+spectrometry&rft.jtitle=Journal+of+Chromatography+A&rft.aulast=Cacho%2C+J.I.%3B+Campillo%2C+N.%3B+Vi%C3%B1as%2C+P.+et+al.&rft.au=Cacho%2C+J.I.%3B+Campillo%2C+N.%3B+Vi%C3%B1as%2C+P.+et+al.&rft.date=2015&rft.volume=1399&rft.pages=18%E2%80%9324&rft_id=info:doi\/10.1016%2Fj.chroma.2015.04.041&rft_id=info:pmid\/25958092&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Jele.C5.84Anal15-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Jele.C5.84Anal15_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jele\u0144, H.H.; Gracka, A. (2015). \"Analysis of black pepper volatiles by solid phase microextraction-gas chromatography: A comparison of terpenes profiles with hydrodistillation\". <i>Journal of Chromatography A<\/i> <b>1418<\/b>: 200\u2013209. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.chroma.2015.09.065\">10.1016\/j.chroma.2015.09.065<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26427328\">26427328<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Analysis+of+black+pepper+volatiles+by+solid+phase+microextraction-gas+chromatography%3A+A+comparison+of+terpenes+profiles+with+hydrodistillation&rft.jtitle=Journal+of+Chromatography+A&rft.aulast=Jele%C5%84%2C+H.H.%3B+Gracka%2C+A.&rft.au=Jele%C5%84%2C+H.H.%3B+Gracka%2C+A.&rft.date=2015&rft.volume=1418&rft.pages=200%E2%80%93209&rft_id=info:doi\/10.1016%2Fj.chroma.2015.09.065&rft_id=info:pmid\/26427328&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StenersonAnal17-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-StenersonAnal17_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-StenersonAnal17_10-1\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-StenersonAnal17_10-2\">10.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stenerson, K.K.; Halpenny, M.R. (2017). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.chromatographyonline.com\/view\/analysis-terpenes-cannabis-using-headspace-solid-phase-microextraction-and-gc-ms\">\"Analysis of Terpenes in Cannabis Using Headspace Solid-Phase Microextraction and GC\u2013MS\"<\/a>. <i>LCGC<\/i> <b>35<\/b> (5): 27\u201332<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.chromatographyonline.com\/view\/analysis-terpenes-cannabis-using-headspace-solid-phase-microextraction-and-gc-ms\">https:\/\/www.chromatographyonline.com\/view\/analysis-terpenes-cannabis-using-headspace-solid-phase-microextraction-and-gc-ms<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Analysis+of+Terpenes+in+Cannabis+Using+Headspace+Solid-Phase+Microextraction+and+GC%E2%80%93MS&rft.jtitle=LCGC&rft.aulast=Stenerson%2C+K.K.%3B+Halpenny%2C+M.R.&rft.au=Stenerson%2C+K.K.%3B+Halpenny%2C+M.R.&rft.date=2017&rft.volume=35&rft.issue=5&rft.pages=27%E2%80%9332&rft_id=https%3A%2F%2Fwww.chromatographyonline.com%2Fview%2Fanalysis-terpenes-cannabis-using-headspace-solid-phase-microextraction-and-gc-ms&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrownValid19-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BrownValid19_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-BrownValid19_11-1\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-BrownValid19_11-2\">11.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brown, A.K.; Xia, Z.; Bulloch, P. et al. (2019). \"Validated quantitative cannabis profiling for Canadian regulatory compliance - Cannabinoids, aflatoxins, and terpenes\". <i>Analytica Chimica Acta<\/i> <b>1088<\/b>: 79\u201388. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.aca.2019.08.042\">10.1016\/j.aca.2019.08.042<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31623719\">31623719<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Validated+quantitative+cannabis+profiling+for+Canadian+regulatory+compliance+-+Cannabinoids%2C+aflatoxins%2C+and+terpenes&rft.jtitle=Analytica+Chimica+Acta&rft.aulast=Brown%2C+A.K.%3B+Xia%2C+Z.%3B+Bulloch%2C+P.+et+al.&rft.au=Brown%2C+A.K.%3B+Xia%2C+Z.%3B+Bulloch%2C+P.+et+al.&rft.date=2019&rft.volume=1088&rft.pages=79%E2%80%9388&rft_id=info:doi\/10.1016%2Fj.aca.2019.08.042&rft_id=info:pmid\/31623719&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IbrahimAnal19-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-IbrahimAnal19_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-IbrahimAnal19_12-1\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-IbrahimAnal19_12-2\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-IbrahimAnal19_12-3\">12.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ibrahim, E.A.; Wang, M.; Radwan, M.M. et al. (2019). \"Analysis of Terpenes in Cannabis sativa L. Using GC\/MS: Method Development, Validation, and Application\". <i>Planta Medica<\/i> <b>85<\/b> (5): 431\u201338. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1055%2Fa-0828-8387\">10.1055\/a-0828-8387<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30646402\">30646402<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Analysis+of+Terpenes+in+Cannabis+sativa+L.+Using+GC%2FMS%3A+Method+Development%2C+Validation%2C+and+Application&rft.jtitle=Planta+Medica&rft.aulast=Ibrahim%2C+E.A.%3B+Wang%2C+M.%3B+Radwan%2C+M.M.+et+al.&rft.au=Ibrahim%2C+E.A.%3B+Wang%2C+M.%3B+Radwan%2C+M.M.+et+al.&rft.date=2019&rft.volume=85&rft.issue=5&rft.pages=431%E2%80%9338&rft_id=info:doi\/10.1055%2Fa-0828-8387&rft_id=info:pmid\/30646402&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShapiraTandem19-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShapiraTandem19_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShapiraTandem19_13-1\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-ShapiraTandem19_13-2\">13.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shapira, A.; Berman, P. Futoran, K. et al. (2019). \"Tandem Mass Spectrometric Quantification of 93 Terpenoids in <i>Cannabis<\/i> Using Static Headspace Injections\". <i>Analytical Chemistry<\/i> <b>91<\/b> (17): 11425\u201332. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Facs.analchem.9b02844\">10.1021\/acs.analchem.9b02844<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31369251\">31369251<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Tandem+Mass+Spectrometric+Quantification+of+93+Terpenoids+in+%27%27Cannabis%27%27+Using+Static+Headspace+Injections&rft.jtitle=Analytical+Chemistry&rft.aulast=Shapira%2C+A.%3B+Berman%2C+P.+Futoran%2C+K.+et+al.&rft.au=Shapira%2C+A.%3B+Berman%2C+P.+Futoran%2C+K.+et+al.&rft.date=2019&rft.volume=91&rft.issue=17&rft.pages=11425%E2%80%9332&rft_id=info:doi\/10.1021%2Facs.analchem.9b02844&rft_id=info:pmid\/31369251&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BakroSimul20-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BakroSimul20_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-BakroSimul20_14-1\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-BakroSimul20_14-2\">14.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bakro, F.; Jedryczka, M.; Wielgusz, K. et al. (2020). \"Simultaneous determination of terpenes and cannabidiol in hemp (<i>Cannabis sativa<\/i> L.) by fast gas chromatography with flame ionization detection\". <i>Journal of Separation Science<\/i> <b>43<\/b> (14): 2817\u201326. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fjssc.201900822\">10.1002\/jssc.201900822<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Simultaneous+determination+of+terpenes+and+cannabidiol+in+hemp+%28%27%27Cannabis+sativa%27%27+L.%29+by+fast+gas+chromatography+with+flame+ionization+detection&rft.jtitle=Journal+of+Separation+Science&rft.aulast=Bakro%2C+F.%3B+Jedryczka%2C+M.%3B+Wielgusz%2C+K.+et+al.&rft.au=Bakro%2C+F.%3B+Jedryczka%2C+M.%3B+Wielgusz%2C+K.+et+al.&rft.date=2020&rft.volume=43&rft.issue=14&rft.pages=2817%E2%80%9326&rft_id=info:doi\/10.1002%2Fjssc.201900822&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GaggiottiPiezoelectric20-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GaggiottiPiezoelectric20_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-GaggiottiPiezoelectric20_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-GaggiottiPiezoelectric20_15-2\">15.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gaggiotti, S.; Palmieri, S.; Pelle, F.D. et al. (2020). \"Piezoelectric peptide-hpDNA based electronic nose for the detection of terpenes; Evaluation of the aroma profile in different Cannabis sativa L. (hemp) samples\". <i>Sensors and Actuators B: Chemical<\/i> <b>308<\/b>: 127697. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.snb.2020.127697\">10.1016\/j.snb.2020.127697<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Piezoelectric+peptide-hpDNA+based+electronic+nose+for+the+detection+of+terpenes%3B+Evaluation+of+the+aroma+profile+in+different+Cannabis+sativa+L.+%28hemp%29+samples&rft.jtitle=Sensors+and+Actuators+B%3A+Chemical&rft.aulast=Gaggiotti%2C+S.%3B+Palmieri%2C+S.%3B+Pelle%2C+F.D.+et+al.&rft.au=Gaggiotti%2C+S.%3B+Palmieri%2C+S.%3B+Pelle%2C+F.D.+et+al.&rft.date=2020&rft.volume=308&rft.pages=127697&rft_id=info:doi\/10.1016%2Fj.snb.2020.127697&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mu.C3.B1oz-RedondoMulti20-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Mu.C3.B1oz-RedondoMulti20_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mu\u00f1oz-Redondo, J.M.; Ruiz-Moreno, M.J.; Puertas, B. et al. (2020). \"Multivariate optimization of headspace solid-phase microextraction coupled to gas chromatography-mass spectrometry for the analysis of terpenoids in sparkling wines\". <i>Talanta<\/i> <b>208<\/b>: 120483. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/110.1016%2Fj.talanta.2019.120483\">110.1016\/j.talanta.2019.120483<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multivariate+optimization+of+headspace+solid-phase+microextraction+coupled+to+gas+chromatography-mass+spectrometry+for+the+analysis+of+terpenoids+in+sparkling+wines&rft.jtitle=Talanta&rft.aulast=Mu%C3%B1oz-Redondo%2C+J.M.%3B+Ruiz-Moreno%2C+M.J.%3B+Puertas%2C+B.+et+al.&rft.au=Mu%C3%B1oz-Redondo%2C+J.M.%3B+Ruiz-Moreno%2C+M.J.%3B+Puertas%2C+B.+et+al.&rft.date=2020&rft.volume=208&rft.pages=120483&rft_id=info:doi\/110.1016%2Fj.talanta.2019.120483&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NguyenQuant20-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NguyenQuant20_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-NguyenQuant20_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nguyen, T.-D.; Riordan-Short, S.; Dang, T.-T.T. et al. (2020). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7081649\">\"Quantitation of Select Terpenes\/Terpenoids and Nicotine Using Gas Chromatography-Mass Spectrometry with High-Temperature Headspace Sampling\"<\/a>. <i>ACS Omega<\/i> <b>5<\/b> (10): 5565-5573. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Facsomega.0c00384\">10.1021\/acsomega.0c00384<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7081649\">PMC7081649<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32201850\">32201850<\/a><span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7081649\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7081649<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quantitation+of+Select+Terpenes%2FTerpenoids+and+Nicotine+Using+Gas+Chromatography-Mass+Spectrometry+with+High-Temperature+Headspace+Sampling&rft.jtitle=ACS+Omega&rft.aulast=Nguyen%2C+T.-D.%3B+Riordan-Short%2C+S.%3B+Dang%2C+T.-T.T.+et+al.&rft.au=Nguyen%2C+T.-D.%3B+Riordan-Short%2C+S.%3B+Dang%2C+T.-T.T.+et+al.&rft.date=2020&rft.volume=5&rft.issue=10&rft.pages=5565-5573&rft_id=info:doi\/10.1021%2Facsomega.0c00384&rft_id=info:pmc\/PMC7081649&rft_id=info:pmid\/32201850&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7081649&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TernelliInnov20-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TernelliInnov20_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-TernelliInnov20_18-1\">18.1<\/a><\/sup> <sup><a href=\"#cite_ref-TernelliInnov20_18-2\">18.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ternelli, M.; Brighenti, V.; Anceschi, L. et al. (2020). \"Innovative methods for the preparation of medical Cannabis oils with a high content of both cannabinoids and terpenes\". <i>Journal of Pharmaceutical and Biomedical Analysis<\/i> <b>186<\/b>: 113296. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpba.2020.113296\">10.1016\/j.jpba.2020.113296<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32334134\">32334134<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Innovative+methods+for+the+preparation+of+medical+Cannabis+oils+with+a+high+content+of+both+cannabinoids+and+terpenes&rft.jtitle=Journal+of+Pharmaceutical+and+Biomedical+Analysis&rft.aulast=Ternelli%2C+M.%3B+Brighenti%2C+V.%3B+Anceschi%2C+L.+et+al.&rft.au=Ternelli%2C+M.%3B+Brighenti%2C+V.%3B+Anceschi%2C+L.+et+al.&rft.date=2020&rft.volume=186&rft.pages=113296&rft_id=info:doi\/10.1016%2Fj.jpba.2020.113296&rft_id=info:pmid\/32334134&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhangOptim20-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhangOptim20_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhang, X.; Wang, C.; Wang, L. et al. (2020). \"Optimization and validation of a head space solid-phase microextraction-arrow gas chromatography-mass spectrometry method using central composite design for determination of aroma compounds in Chinese liquor (Baijiu)\". <i>Journal of Chromatography A<\/i> <b>1610<\/b>: 460584. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.chroma.2019.460584\">10.1016\/j.chroma.2019.460584<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31607446\">31607446<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Optimization+and+validation+of+a+head+space+solid-phase+microextraction-arrow+gas+chromatography-mass+spectrometry+method+using+central+composite+design+for+determination+of+aroma+compounds+in+Chinese+liquor+%28Baijiu%29&rft.jtitle=Journal+of+Chromatography+A&rft.aulast=Zhang%2C+X.%3B+Wang%2C+C.%3B+Wang%2C+L.+et+al.&rft.au=Zhang%2C+X.%3B+Wang%2C+C.%3B+Wang%2C+L.+et+al.&rft.date=2020&rft.volume=1610&rft.pages=460584&rft_id=info:doi\/10.1016%2Fj.chroma.2019.460584&rft_id=info:pmid\/31607446&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RaposoChall21-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RaposoChall21_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-RaposoChall21_20-1\">20.1<\/a><\/sup> <sup><a href=\"#cite_ref-RaposoChall21_20-2\">20.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Raposo, F.; Barcel\u00f3, D. (2021). \"Challenges and strategies of matrix effects using chromatography-mass spectrometry: An overview from research versus regulatory viewpoints\". <i>TrAC Trends in Analytical Chemistry<\/i> <b>134<\/b>: 116068. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.trac.2020.116068\">10.1016\/j.trac.2020.116068<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Challenges+and+strategies+of+matrix+effects+using+chromatography-mass+spectrometry%3A+An+overview+from+research+versus+regulatory+viewpoints&rft.jtitle=TrAC+Trends+in+Analytical+Chemistry&rft.aulast=Raposo%2C+F.%3B+Barcel%C3%B3%2C+D.&rft.au=Raposo%2C+F.%3B+Barcel%C3%B3%2C+D.&rft.date=2021&rft.volume=134&rft.pages=116068&rft_id=info:doi\/10.1016%2Fj.trac.2020.116068&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LigorComp14-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LigorComp14_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-LigorComp14_21-1\">21.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ligor, M.; Stankevi\u010dius, M.; Wenda-Piesik, A. et al. (2014). \"Comparative Gas Chromatographic\u2013Mass Spectrometric Evaluation of Hop (Humulus lupulus L.) Essential Oils and Extracts Obtained Using Different Sample Preparation Methods\". <i>Food Analytical Methods<\/i> <b>7<\/b>: 1433\u201342. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs12161-013-9767-5\">10.1007\/s12161-013-9767-5<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comparative+Gas+Chromatographic%E2%80%93Mass+Spectrometric+Evaluation+of+Hop+%28Humulus+lupulus+L.%29+Essential+Oils+and+Extracts+Obtained+Using+Different+Sample+Preparation+Methods&rft.jtitle=Food+Analytical+Methods&rft.aulast=Ligor%2C+M.%3B+Stankevi%C4%8Dius%2C+M.%3B+Wenda-Piesik%2C+A.+et+al.&rft.au=Ligor%2C+M.%3B+Stankevi%C4%8Dius%2C+M.%3B+Wenda-Piesik%2C+A.+et+al.&rft.date=2014&rft.volume=7&rft.pages=1433%E2%80%9342&rft_id=info:doi\/10.1007%2Fs12161-013-9767-5&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChiesaAccel17-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ChiesaAccel17_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-ChiesaAccel17_22-1\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chiesa, L.M.; Labella, G.F.; Panseri, S. et al. (2017). \"Accelerated solvent extraction by using an 'in-line' clean-up approach for multiresidue analysis of pesticides in organic honey\". <i>Food Additives & Contaminants: Part A<\/i> <b>34<\/b> (5): 809\u201318. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F19440049.2017.1292558\">10.1080\/19440049.2017.1292558<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28277176\">28277176<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Accelerated+solvent+extraction+by+using+an+%27in-line%27+clean-up+approach+for+multiresidue+analysis+of+pesticides+in+organic+honey&rft.jtitle=Food+Additives+%26+Contaminants%3A+Part+A&rft.aulast=Chiesa%2C+L.M.%3B+Labella%2C+G.F.%3B+Panseri%2C+S.+et+al.&rft.au=Chiesa%2C+L.M.%3B+Labella%2C+G.F.%3B+Panseri%2C+S.+et+al.&rft.date=2017&rft.volume=34&rft.issue=5&rft.pages=809%E2%80%9318&rft_id=info:doi\/10.1080%2F19440049.2017.1292558&rft_id=info:pmid\/28277176&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuSimult20-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HuSimult20_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-HuSimult20_23-1\">23.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hu, A.; Qiu, M.; Liu, H. et al. (2020). \"Simultaneous determination of phthalate diesters and monoesters in soil using accelerated solvent extraction and ultra-performance liquid chromatography coupled with tandem mass spectrometry\". <i>Journal of Chromatography A<\/i> <b>1626<\/b>: 104366. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.microc.2019.104366\">10.1016\/j.microc.2019.104366<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Simultaneous+determination+of+phthalate+diesters+and+monoesters+in+soil+using+accelerated+solvent+extraction+and+ultra-performance+liquid+chromatography+coupled+with+tandem+mass+spectrometry&rft.jtitle=Journal+of+Chromatography+A&rft.aulast=Hu%2C+A.%3B+Qiu%2C+M.%3B+Liu%2C+H.+et+al.&rft.au=Hu%2C+A.%3B+Qiu%2C+M.%3B+Liu%2C+H.+et+al.&rft.date=2020&rft.volume=1626&rft.pages=104366&rft_id=info:doi\/10.1016%2Fj.microc.2019.104366&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NingAccel20-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NingAccel20_24-0\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-NingAccel20_24-1\">24.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Min, N.; Yao, J.; Amde, M. et al. (2020). \"Accelerated solvent extraction combined with GC\u2013MS: A convenient technique for the determination and compound-specific stable isotope analysis of phthalates in mine tailings\". <i>Microchemical Journal<\/i> <b>153<\/b>: 461347. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.chroma.2020.461347\">10.1016\/j.chroma.2020.461347<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32797827\">32797827<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Accelerated+solvent+extraction+combined+with+GC%E2%80%93MS%3A+A+convenient+technique+for+the+determination+and+compound-specific+stable+isotope+analysis+of+phthalates+in+mine+tailings&rft.jtitle=Microchemical+Journal&rft.aulast=Min%2C+N.%3B+Yao%2C+J.%3B+Amde%2C+M.+et+al.&rft.au=Min%2C+N.%3B+Yao%2C+J.%3B+Amde%2C+M.+et+al.&rft.date=2020&rft.volume=153&rft.pages=461347&rft_id=info:doi\/10.1016%2Fj.chroma.2020.461347&rft_id=info:pmid\/32797827&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HerringtonHunting20-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HerringtonHunting20_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Herrington, J.S.; G\u00f3mez-R\u00edos, G.A.; Myers, C. et al. (2020). \"Hunting Molecules in Complex Matrices with SPME Arrows: A Review\". <i>Separations<\/i> <b>7<\/b> (1): 12. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fseparations7010012\">10.3390\/separations7010012<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hunting+Molecules+in+Complex+Matrices+with+SPME+Arrows%3A+A+Review&rft.jtitle=Separations&rft.aulast=Herrington%2C+J.S.%3B+G%C3%B3mez-R%C3%ADos%2C+G.A.%3B+Myers%2C+C.+et+al.&rft.au=Herrington%2C+J.S.%3B+G%C3%B3mez-R%C3%ADos%2C+G.A.%3B+Myers%2C+C.+et+al.&rft.date=2020&rft.volume=7&rft.issue=1&rft.pages=12&rft_id=info:doi\/10.3390%2Fseparations7010012&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrillAHigh20-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrillAHigh20_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krill, C.; Rochfort, S.; Spangenberg, G. (2020). \"A High-Throughput Method for the Comprehensive Analysis of Terpenes and Terpenoids in Medicinal Cannabis Biomass\". <i>Metabolites<\/i> <b>10<\/b> (7): 276. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fmetabo10070276\">10.3390\/metabo10070276<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+High-Throughput+Method+for+the+Comprehensive+Analysis+of+Terpenes+and+Terpenoids+in+Medicinal+Cannabis+Biomass&rft.jtitle=Metabolites&rft.aulast=Krill%2C+C.%3B+Rochfort%2C+S.%3B+Spangenberg%2C+G.&rft.au=Krill%2C+C.%3B+Rochfort%2C+S.%3B+Spangenberg%2C+G.&rft.date=2020&rft.volume=10&rft.issue=7&rft.pages=276&rft_id=info:doi\/10.3390%2Fmetabo10070276&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RestekFast20-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RestekFast20_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.restek.com\/en\/technical-literature-library\/articles\/fast-low-solvent-analysis-of-cannabinoids-increases-lab-productivity-and-decreases-solvent-costs\/\">\"Fast, Low-Solvent Analysis of Cannabinoids Increases Lab Productivity and Decreases Solvent Costs\"<\/a>. Restek. 2020<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.restek.com\/en\/technical-literature-library\/articles\/fast-low-solvent-analysis-of-cannabinoids-increases-lab-productivity-and-decreases-solvent-costs\/\">https:\/\/www.restek.com\/en\/technical-literature-library\/articles\/fast-low-solvent-analysis-of-cannabinoids-increases-lab-productivity-and-decreases-solvent-costs\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 April 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Fast%2C+Low-Solvent+Analysis+of+Cannabinoids+Increases+Lab+Productivity+and+Decreases+Solvent+Costs&rft.atitle=&rft.date=2020&rft.pub=Restek&rft_id=https%3A%2F%2Fwww.restek.com%2Fen%2Ftechnical-literature-library%2Farticles%2Ffast-low-solvent-analysis-of-cannabinoids-increases-lab-productivity-and-decreases-solvent-costs%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChambersSystem07-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChambersSystem07_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chambers, E.; Wagrowski-Diehl, D.M.; Lu, Z. et al. (2007). \"Systematic and comprehensive strategy for reducing matrix effects in LC\/MS\/MS analyses\". <i>Journal of Chromatography B<\/i> <b>852<\/b> (1\u20132): 22\u201334. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jchromb.2006.12.030\">10.1016\/j.jchromb.2006.12.030<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Systematic+and+comprehensive+strategy+for+reducing+matrix+effects+in+LC%2FMS%2FMS+analyses&rft.jtitle=Journal+of+Chromatography+B&rft.aulast=Chambers%2C+E.%3B+Wagrowski-Diehl%2C+D.M.%3B+Lu%2C+Z.+et+al.&rft.au=Chambers%2C+E.%3B+Wagrowski-Diehl%2C+D.M.%3B+Lu%2C+Z.+et+al.&rft.date=2007&rft.volume=852&rft.issue=1%E2%80%932&rft.pages=22%E2%80%9334&rft_id=info:doi\/10.1016%2Fj.jchromb.2006.12.030&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Hajslov.C3.A1Matrix03-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Hajslov.C3.A1Matrix03_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hajslov\u00e1, J.; Zrostl\u00edkov\u00e1, J. (2003). \"Matrix effects in (ultra)trace analysis of pesticide residues in food and biotic matrices\". <i>Journal of Chromatography A<\/i> <b>1000<\/b> (1\u20132): 181\u201397. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fs0021-9673%2803%2900539-9\">10.1016\/s0021-9673(03)00539-9<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12877171\">12877171<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Matrix+effects+in+%28ultra%29trace+analysis+of+pesticide+residues+in+food+and+biotic+matrices&rft.jtitle=Journal+of+Chromatography+A&rft.aulast=Hajslov%C3%A1%2C+J.%3B+Zrostl%C3%ADkov%C3%A1%2C+J.&rft.au=Hajslov%C3%A1%2C+J.%3B+Zrostl%C3%ADkov%C3%A1%2C+J.&rft.date=2003&rft.volume=1000&rft.issue=1%E2%80%932&rft.pages=181%E2%80%9397&rft_id=info:doi\/10.1016%2Fs0021-9673%2803%2900539-9&rft_id=info:pmid\/12877171&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CBCC16-42-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CBCC16-42_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/bcc.ca.gov\/law_regs\/cannabis_order_of_adoption.pdf\">\"California Code of Regulations Title 16, Division 42, Bureau of Cannabis Control\"<\/a> (PDF). Bureau of Cannabis Control. 2019<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/bcc.ca.gov\/law_regs\/cannabis_order_of_adoption.pdf\">https:\/\/bcc.ca.gov\/law_regs\/cannabis_order_of_adoption.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 August 2020<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=California+Code+of+Regulations+Title+16%2C+Division+42%2C+Bureau+of+Cannabis+Control&rft.atitle=&rft.date=2019&rft.pub=Bureau+of+Cannabis+Control&rft_id=https%3A%2F%2Fbcc.ca.gov%2Flaw_regs%2Fcannabis_order_of_adoption.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added. The original article lists references in alphabetical order; this wiki organizes them by order of appearance, by design.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150552\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.549 seconds\nReal time usage: 0.581 seconds\nPreprocessor visited node count: 23449\/1000000\nPreprocessor generated node count: 33130\/1000000\nPost\u2010expand include size: 198918\/2097152 bytes\nTemplate argument size: 66218\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 491.588 1 - -total\n 85.41% 419.861 1 - Template:Reflist\n 71.44% 351.197 30 - Template:Citation\/core\n 65.91% 324.006 26 - Template:Cite_journal\n 9.04% 44.455 4 - Template:Cite_web\n 8.26% 40.614 41 - Template:Citation\/identifier\n 7.74% 38.067 1 - Template:Infobox_journal_article\n 7.35% 36.119 1 - Template:Infobox\n 5.05% 24.827 80 - Template:Infobox\/row\n 3.43% 16.870 31 - Template:Citation\/make_link\n-->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis\">https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t<\/body>","a3b638adc3dd9824430235085d4b072b_images":["https:\/\/www.cannaqa.wiki\/images\/6\/6e\/Fig1_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/8\/88\/Tab2_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/7\/76\/Tab3_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/8\/88\/Fig1Act_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/f\/f4\/Fig2_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/d\/d6\/Tab4_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/2\/2b\/Fig3_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/f\/fa\/Fig4_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/5\/59\/Tab5_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/9\/91\/Tab6_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/1\/15\/Tab7_Myers_FrontChem2021_9.jpg","https:\/\/www.cannaqa.wiki\/images\/5\/57\/Fig5_Myers_FrontChem2021_9.jpg"],"a3b638adc3dd9824430235085d4b072b_timestamp":1625497551,"d78732feb44acdd119e6c7a504f8afa4_type":"article","d78732feb44acdd119e6c7a504f8afa4_title":"Metabolomic analysis of cannabinoid and essential oil profiles in different hemp (Cannabis sativa L.) phenotypes (Er\u017een et al. 2021)","d78732feb44acdd119e6c7a504f8afa4_url":"https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes","d78732feb44acdd119e6c7a504f8afa4_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Metabolomic analysis of cannabinoid and essential oil profiles in different hemp (Cannabis sativa L.) phenotypes\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom CannaQAWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nMetabolomic analysis of cannabinoid and essential oil profiles in different hemp (Cannabis sativa L.) phenotypesJournal\n \nPlantsAuthor(s)\n \nEr\u017een, Marjeta; Ko\u0161ir, Iztok J.; Ocvirk, Miha; Kreft, Samo; \u010cerenak, AndrejaAuthor affiliation(s)\n \nSlovenian Institute of Hop Research and Brewing, University of LjubljanaPrimary contact\n \nEmail: andreja dot cerenak at ihps dot siEditors\n \nIorizzi, MariaYear published\n \n2021Volume and issue\n \n10(5)Article #\n \n966DOI\n \n10.3390\/plants10050966ISSN\n \n2223-7747Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/2223-7747\/10\/5\/966\/htmDownload\n \nhttps:\/\/www.mdpi.com\/2223-7747\/10\/5\/966\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Results and discussion \n\n3.1 Differentiation of phenotypes according to visual traits \n3.2 Chemical analysis of the essential oil of hemp (Cannabis sativa L.) \n3.3 Chemical analysis of cannabinoids in hemp (Cannabis sativa L.) phenotypes \n3.4 Chemical analysis of both cannabinoids and essential oil in hemp (Cannabis sativa L.) phenotypes \n\n\n4 Materials and methods \n\n4.1 Plant material \n4.2 Standards and reagents \n4.3 Sample preparation \n4.4 GC\u2013FID analysis \n4.5 HPLC analysis \n4.6 Statistical analyses \n\n\n5 Conclusions \n6 Supplementary materials \n7 Acknowdledgements \n\n7.1 Author contributions \n7.2 Funding \n7.3 Conflicts of interest \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nHemp (Cannabis sativa L.) cannabinoids and terpenoids have therapeutic effects on human and animal health. Cannabis plants can often have a relatively high heterogeneity, which leads to different phenotypes that have different chemical profiles despite being from the same variety. Little information exists about cannabinoid and terpenoid profiles in different hemp phenotypes within the same variety. For this study, 11 phenotypes from three different varieties\u2014Carmagnola Selected (CS), Tiborszallasi (TS), and Finola Selection (FS)\u2014were analyzed. The components of essential oil (29) were analyzed using gas chromatography with flame ionization detection (GC-FID), and 10 different cannabinoids of each phenotype were determined using high-performance liquid chromatography (HPLC). \nPrincipal component analysis (PCA) and analysis of variance (ANOVA) showed that according to the components of essential oil, FS and TS plants were more uniform than CS plants, where there were great differences between CI and CII phenotypes. The content of cannabidiolic acid (CBDA) was the highest in all four FS phenotypes. By comparing cannabinoid profiles, FS was clearly separated from TS and CS, while these two varieties were not clearly distinguishable. Phenotypes TV and CI had the highest total content of tetrahydrocannabinol (\u03949-THC), while all phenotypes of FS had the highest total content of cannabidiol (CBD). The highest total content of cannabigerol (CBG) was determined in phenotype CI. Obtained results are useful for the development of new supplementary ingredients, for different pharmacy treatments, and for further breeding purposes.\nKeywords: Cannabis sativa L., Cannabaceae, cannabinoids, essential oils, terpenes, GC-FID, HPLC\n\nIntroduction \nHemp (Cannabis sativa L.) originated from central Asia and has been used for human and animal food, as a source of fiber for ropes, and in medicine.[1][2] It contains more than 500 phytochemicals with many therapeutic purposes and has been used to treat epilepsy, Alzheimer\u2019s disease, Parkinson\u2019s disease, multiple sclerosis, pain and nausea in cancer patients, diabetes, and eating disorders.[3]\nThe most well-known phytochemicals are secondary metabolites, such as cannabinoids and terpenoids.[4] More than 150 cannabinoids have already been identified in hemp.[5] The most active and studied compounds are \u03949 tetrahydrocannabinol (\u03949-THC), cannabidiol (CBD), cannabigerol (CBG), cannabichromene (CBC), and their carboxylated forms.[3] Terpenoids in essential oil are divided into monoterpenes and sesquiterpenes, which are responsible for hemp fragrance and flavor and also contribute to therapeutic effects. There are generally fewer sesquiterpenes than monoterpenes detected in hemp flowers. The highest content of cannabinoids and terpenoids is found in the glandular trichomes on bracts.[6]\nPrecursors for cannabinoids have two biosynthetic pathways. The polyketide pathway leads to olivetolic acid (OLA), and the plastidial 2-C-methyl-D-erytritol 4-phosphate (MEP) pathway leads to geranyl diphosphate (GPP). Precursors OLA and GPP form cannabigerolic acid (CBGA), which is a precursor for different cannabinoids, as well as tetrahydrocannabinolic acid (THCA), cannabidiolic acid (CBDA), and cannabichromenic acid (CBCA).[7] Terpenoids are composed of isoprene units. Similar to cannabinoids, terpenoids also have different biosynthetic pathways. Sesquiterpenes and triterpenes are formed from the cytosolic mevalonic acid (MVA) pathway, while monoterpenes, diterpenes, and tetraterpenes are formed via the plastid-localized (MEP) pathway. Subsequently, precursors of sesquiterpene farnesyl diphosphate (FPP) and monoterpene geranyl diphosphate are formed.[7]\nTerpenoids and cannabinoids may have a synergistic effect on human and animal health.[8] An example of the positive effects of the combined use of cannabinoids and terpenoids is acne therapy, in which CBD, limonene, linalool, and pinene are involved. Cannabinoids and terpenoids such as CBG and pinene also have a combined effect on MRSA (methicillin-resistant Staphylococcus aureus).[9][10] However, the issue of synergy remains controversial and needs further investigation.\nAccording to the chemical composition, there are five major hemp chemotypes. Small and Beckstead[11] determined three chemotypes: chemotype I, with a THC content higher than 0.3% and CBD content lower than 0.5%; chemotype II (intermediate type), with a THC and CBD ratio that is roughly equal; and chemotype III, with a higher CBD content than 0.5% and THC content lower than 0.3% of the flower dry matter. Later, Fournier et al.[12] determined two other chemotypes: chemotype IV, with a prevalence of CBG higher than 0.3% of the flower dry matter, and chemotype V, with an undetectable content of cannabinoids. \nNumerous scientists have studied species and subspecies of Cannabis.[13] In general, it is known that hemp and marijuana differ based on THC content. Hemp is supposed to have THC content below 0.2\u20131%, which depends on the legislation of different countries, while marijuana could reach THC content up to 20 to 30% in dry inflorescences.[14] In 2015, Sawler et al.[15] determined that hemp and marijuana significantly differ at the genome level, that different marijuana types are often not genetically close, and that THC is not related to the genetic distinction between hemp and marijuana. Hemp has been used for food and fibers, while marijuana was mostly used in traditional medicine.[16] However, marijuana was prohibited and criminalized all around the world due to the psychoactive nature of THC. In 1988, the United Nations Convention against Illicit Traffic in Narcotic Drugs and Psychotropic Substances prohibited the use, production, and cultivation of Cannabis plants, which was recognized as a narcotic drug with psychotropic compounds, still posing a major problem in the legalization of cannabis with higher THC content.[17] Due to many different research studies based on the positive effects of cannabis on numerous diseases, more and more countries are slowly revising their legislation in favor of growing Cannabis plants for medical and scientific purposes in restricted area.\nCannabis varieties (strains) are often not fully inbred; therefore, they have a relatively high level of heterogeneity and instability, compared to other crops. Different phenotypes can be found within one variety. One of the major breeding challenges is that Cannabis plants are mostly dioecious, and they cannot pollinate themselves; hence, outcrossing commonly occurs.[18] This paper aims to clarify the distinction between three different hemp varieties and their 11 phenotypes according to an analysis of cannabinoids and terpenoids. Three varieties\u2014Carmagnola Selected (CS), Tiborszallasi (TS), and Finola Selection (FS)\u2014were chosen because of the expressed interest of our partners from the industry. All of them are registered on the E.U. variety list and are grown as out-growing varieties. Together, we found out that different phenotypes could be detected within them, and we supposed that they could express different chemical profiles as well, with a further different application in pharmacy. That said, the ultimate objective was to establish a connection between the chemical composition and morphological characteristics of hemp plants and to identify phenotypes with an interesting ratio between cannabinoids for further pharmaceutical applications.\n\nResults and discussion \nDifferentiation of phenotypes according to visual traits \nIn total, 11 different phenotypes were selected according to visual traits observed within varieties that were compared with each other, and reference types[19][20][21] (certified types from breeders) were added for each variety (Table 1). However, this is a first preliminary comparison between different phenotypes within mentioned varieties. Photos of all 11 phenotypes are presented in Figures S3\u2013S14 (see Supplementary materials). According to the Slovenian Ministry of Agriculture, Forestry, and Food in 2020, there was 6.67 ha of Carmagnola, 0.86 ha of Tiborszallasi, and 19.77 ha of Finola.\n\n\n\n\n\n\n\nTable 1. The 11 different phenotypes (CI, CII, TI, TII, TIII, TIV, TV, FI, FII, FIII, FIV) that are defined by six different visual traits: size, color, leaf size, inflorescences, anthocyanin coloration of leaf petiole, and branching. For each variety, reference types are added. Note that size is the comparison between the height of plants within phenotypes in each variety. Plant color was described as light, medium, or dark green. Branching is scales, with * meaning little branched, up to **** meaning highly branched. - = No data available.\n\n\nPhenotype\n\nSize\n\nColor\n\nLeaf size\n\nInflorescences\n\nAnthocyanin coloration of leaf petiole\n\nBranching\n\nRemarks\n\n\nCarmagnola Selected\n\n\nCI\n\nTall\n\nLight\n\nLarge\n\nSmall\n\nNo\n\n***\n\n\n\n\nCII\n\nTall\n\nDark\n\nSmall\n\nSmall\n\nYes\n\n****\n\n\n\n\nReference type\n\nTall\n\nDark\n\nMedium\n\n-\n\nMedium\n\n-\n\n\n\n\nTiborszallasi\n\n\nTI\n\nTall\n\nMedium\n\nMedium\n\nSmall\n\nNo\n\n****\n\n\n\n\nTII\n\nMedium\n\nDark\n\nMedium\n\nMedium\n\nYes\n\n***\n\n\n\n\nTIII\n\nSmall\n\nDark\n\nSmall\n\nMedium\n\nYes\n\n**\n\n\n\n\nTIV\n\nMedium\n\nDark\n\nLarge\n\nBig\n\nNo\n\n***\n\nCompact flowers\n\n\nTV\n\nSmall\n\nMedium\n\nSmall\n\nMedium\n\nYes\n\n*\n\nStrong anthocyanin coloration of the whole plant\n\n\nReference type\n\nTall\n\nDark\n\n-\n\n-\n\n-\n\n****\n\n\n\n\nFinola Selection\n\n\nFI\n\nTall\n\nDark\n\nMedium\n\nBig\n\nNo\n\n****\n\n\n\n\nFII\n\nMedium\n\nMedium\n\nMedium\n\nBig\n\nNo\n\n***\n\n\n\n\nFIII\n\nMedium\n\nLight\n\nMedium\n\nMedium\n\nYes\n\n****\n\n\n\n\nFIV\n\nMedium\n\nDark\n\nMedium\n\nBig\n\nYes\n\n****\n\n\n\n\nReference type\n\nSmall\n\nMedium\n\nSmall-medium\n\n-\n\nNo\n\n***\n\n\n\n\n\nFive plants with the same observed traits within one phenotype were labeled and separately sampled. Hemp is an open-pollinated plant and, therefore, also more prone to non-uniformity. Pollen can disperse a few kilometers in relation to the wind direction[22], which could be one of the reasons for higher heterogeneity in Cannabis plant varieties. Additionally, newly bred populations and marijuana populations are more uniform and can be easily grouped by their desirable traits, such as high THC\/CBD level, high limonene, or other terpenoid levels.[15]\n\nChemical analysis of the essential oil of hemp (Cannabis sativa L.) \nBriefly, the 11 phenotypes from CS, TS, and FS contained 0.09\u20133.38 mL of essential oil per 100 g of air-dried flower (1.34 mL\/100 g, on average). FS achieved the highest average content of essential oil (2.81 mL\/100 g air-dried flower), compared to CS (0.38 mL\/100 g air-dried flower) and TS (0.54 mL\/100 g air-dried flower). The greatest relative difference between phenotypes within varieties was between CS phenotypes (p < 0.0001) in relation to essential oil. CI achieved 0.23 mL\/100 g of air-dried flower, while CII achieved 0.53 mL\/100 g air dry flower of essential oil. When comparing monoecious and dioecious varieties, we noted that Bertoli et al.[6] discovered a higher content of essential oil in dioecious plants. Nissen et al.[23] also reported about low content of essential oil in the CS variety. Significant differences have been recorded between different hemp varieties according to essential oil and cannabinoid content in previous studies as well.[24][25]\nAs expected, the most abundant terpenoids were myrcene, \u03b2-caryophyllene, \u03b1-pinene, and \u03b1-humulene in all three varieties.[3][6][23] The proportion of the 10 main components of essential oil were compared with analysis of variance (ANOVA) and presented in superscripts in Table 2 (as well as Table S1, Supplementary materials), in which all 29 analyzed components are presented. Notable points about these analyzed components include:\n\n According to ANOVA, phenotypes CI and CII showed significant differences based on \u03b1-pinene, \u03b2-pinene, 3-Carene, terpinolene, \u03b2-caryophyllene, \u03b1-humulene, caryophyllene oxide, \u03b2-eudesmol, and phytol. The CII phenotype had significantly higher contents of all main monoterpenes (\u03b1-pinene, \u03b2-pinene, 3-carene, and terpinolene) than phenotype CI, while the contents of all main sesquiterpenes (\u03b2-caryophyllene, \u03b1-humulene, caryophyllene oxide, \u03b2-eudesmol) and phytol were significantly higher in the CI phenotype. \n When comparing phenotypes of TS, differences were shown in caryophyllene oxide. \n Terpinolene and \u03b1-terpinene were found at different concentrations in FS phenotypes. As is evident from Table 2, the phenotypes of FS had very high proportions of limonene (4.1\u20135.2%), in comparison to CS and TS phenotypes, which is of great value since FS phenotypes also reached the highest content of essential oil. The lowest proportion of limonene was in the TI phenotype (0.6%). \n Phenotypes of FS had the highest proportions of \u03b1-terpineol, \u03b2-eudesmol, and \u03b1-bisabolol and very low proportions of caryophyllene, oxide, and undetachable proportion of phytol, cis-nerolidol, etc. \n The highest proportion of \u03b1-pinene was in all TS phenotypes, especially in TIV (11.9%), and in phenotype CII (11.9%). A distinctly lower proportion was identified in FIV (0.7%). \n The highest proportion of myrcene was also observed in the five TS phenotypes; TI had the highest proportion among the TS phenotypes (29.9%). \n The highest proportion of, \u03b2-pinene, 3-carene, \u03b3-terpinene, terpinolene, borneol, and menthol occurred in CII among all phenotypes, while the highest proportion of \u03b1-cedrene, \u03b2-caryophyllene, \u03b1-humulene, cis-nerolidol, geranyl isobutyrate, caryophyllene oxide, \u03b2-eudesmol, and phytol occurred in CI (almost all components of sesquiterpenes). However, the CS variety also had the lowest amount of essential oil compared to other investigated varieties. \n FIII had the highest proportion of \u03b1-terpinene (0.6%) and fenchone (0.2%), and TI had the highest proportion of p-cymene (0.1%). \n The highest proportion of borneol (0.2%) and geranyl acetate (0.3%) was in FIV. FII contained the most \u03b1-terpineol (1.0%). \n The terpenoids p-cymene, camphor, isoborneol, \u03b2-citronellol, and neryl acetate had contents lower than 0.1% or were undetectable.\n\r\n\n\n\n\n\n\n\n\nTable 2. Average essential oil (EO) content (mL\/100 g air-dried flower) of the main components in the inflorescence and average composition (%) of essential oil. Groups (a, b, c, and d) were determined by analysis of variance (ANOVA) from different phenotypes of Carmagnola Selected, Tiborszallasi, and Finola Selection. The same letters present similarities between phenotypes, while different letters represent differences between phenotypes according to components of essential oil. The mean \u00b1 standard deviation (SD) is reported.\n\n\nPhenotype\n\nCI\n\nSD\n\nCII\n\nSD\n\nTI\n\nSD\n\nTII\n\nSD\n\nTIII\n\nSD\n\nTIV\n\nSD\n\nTV\n\nSD\n\nFI\n\nSD\n\nFII\n\nSD\n\nFIII\n\nSD\n\nFIV\n\nSD\n\n\nAverage EO content\n\n0.23\n\n0.10\n\n0.53\n\n0.16\n\n0.58\n\n0.23\n\n0.39\n\n0.04\n\n0.52\n\n0.15\n\n0.64\n\n0.22\n\n0.56\n\n0.17\n\n2.75\n\n0.33\n\n3.11\n\n0.23\n\n2.82\n\n0.41\n\n2.59\n\n0.38\n\n\n\u03b1- Pinene\n\n2.5\r\na,b\n\n1.7\n\n11.6\r\nc,d\n\n4.6\n\n10.3\r\nb,c,d\n\n3.3\n\n7.4\r\na,b,c,d\n\n3.7\n\n8.3\r\na,b,c,d\n\n5.2\n\n11.9\r\nd\n\n6.3\n\n11.5\r\nc,d\n\n12.2\n\n3.7\r\na,b,c\n\n4.1\n\n6.0\r\na,b,c,d\n\n3.4\n\n3.5\r\na,b\n\n3.4\n\n0.7\r\na\n\n0.4\n\n\n\u03b2-Pinene\n\n2.1\r\na,b\n\n0.9\n\n7.0\r\nc\n\n3.3\n\n3.9\r\na,b\n\n0.4\n\n3.3\r\na,b\n\n1.3\n\n3.5\r\na,b\n\n1.9\n\n4.5\r\na,b\n\n2.1\n\n4.5\r\na,b\n\n3.6\n\n2.3\r\na,b\n\n1.5\n\n3.6\r\na,b\n\n0.9\n\n2.8\r\na,b\n\n1.0\n\n1.3\r\na\n\n0.7\n\n\nMyrcene\n\n10.5\r\na\n\n7.0\n\n20.9\r\na,b,c\n\n13.0\n\n29.9\r\nc\n\n4.4\n\n21.4\r\na,b,c\n\n8.9\n\n25.2\r\nb,c\n\n8.1\n\n26.7\r\nb,c\n\n7.2\n\n24.9\r\nb,c\n\n9.3\n\n19.2\r\na,b,c\n\n12.3\n\n21.7\r\na,b,c\n\n10.7\n\n16.0\r\na,b\n\n7.4\n\n16.1\r\na,b\n\n7.6\n\n\nLimonene\n\n2.6\r\na,b,c\n\n2.5\n\n3.3\r\na,b,c\n\n2.6\n\n0.6\r\na\n\n0.4\n\n1.8\r\na,b\n\n3.1\n\n1.0\r\na\n\n1.4\n\n1.7\r\na,b\n\n3.0\n\n1.2\r\na\n\n1.1\n\n4.1\r\nb,c\n\n0.9\n\n4.4\r\nc\n\n0.6\n\n4.5\r\nc\n\n1.6\n\n5.2\r\nc\n\n0.7\n\n\nTerpinolene\n\n2.9\r\na\n\n1.8\n\n13.4\r\nb,c\n\n8.3\n\n2.7\r\na\n\n4.1\n\n2.8\r\na\n\n3.7\n\n2.6\r\na\n\n3.1\n\n3.8\r\na,b\n\n5.1\n\n4.2\r\na,b\n\n4.9\n\n5.5\r\na,b\n\n7.1\n\n10.6\r\na,b,c\n\n9.6\n\n16.0\r\nc\n\n9.1\n\n7.9\r\na,b,c\n\n10.2\n\n\n\u03b2-Caryophyllene\n\n21.3\r\nc\n\n6.9\n\n10.5\r\na\n\n3.0\n\n10.8\r\na\n\n4.4\n\n16.1\r\na,b,c\n\n8.1\n\n16.3\r\na,b,c\n\n4.4\n\n11.9\r\na\n\n7.7\n\n16.1\r\na,b,c\n\n7.9\n\n19.7\r\nb,c\n\n5.6\n\n12.5\r\na,b\n\n3.6\n\n13.6\r\na,b\n\n3.1\n\n19.7\r\nb,c\n\n6.9\n\n\n\u03b1-Humulene\n\n8.8\r\nb\n\n2.8\n\n4.5\r\na\n\n1.4\n\n4.0\r\na\n\n2.2\n\n5.6\r\na\n\n2.6\n\n5.7\r\na\n\n1.8\n\n4.1\r\na\n\n2.5\n\n5.0\r\na\n\n2.4\n\n6.4\r\na,b\n\n2.1\n\n3.9\r\na\n\n1.1\n\n4.2\r\na\n\n1.0\n\n6.7\r\na,b\n\n2.7\n\n\nCaryophyllene oxide\n\n2.9\r\nd\n\n0.8\n\n1.1\r\nb\n\n0.9\n\n1.5\r\nb,c\n\n0.4\n\n2.6\r\nd\n\n0.9\n\n2.2\r\nc,d\n\n0.8\n\n1.6\r\nb,c\n\n0.5\n\n1.3\r\nb\n\n0.5\n\n0.3\r\na\n\n0.2\n\n0.3\r\na\n\n0.1\n\n0.3\r\na\n\n0.1\n\n0.4\r\na\n\n0.2\n\n\n\u03b2-Eudesmol\n\n2.8\r\nb\n\n1.9\n\n0.9\r\na\n\n1.5\n\n0.5\r\na\n\n0.6\n\n0.8\r\na\n\n0.8\n\n1.0\r\na\n\n1.5\n\n0.8\r\na\n\n0.8\n\n0.4\r\na\n\n0.4\n\n1.3\r\na\n\n0.22\n\n1.31\r\na\n\n0.24\n\n1.09\r\na\n\n0.47\n\n1.41\r\na\n\n0.24\n\n\n\u03b1-Bisabolol\n\n0.9\r\na\n\n0.3\n\n0.4\r\na\n\n0.4\n\n0.7\r\na\n\n0.6\n\n0.4\r\na\n\n0.3\n\n0.8\r\na\n\n0.8\n\n0.6\r\na\n\n0.4\n\n0.2\r\na\n\n0.1\n\n4.8\r\nd\n\n1.15\n\n3.03\r\nb\n\n1.67\n\n3.37\r\nb,c\n\n0.90\n\n4.31\r\nc,d\n\n0.26\n\n\n\nTerpenoids have a great impact on human health.[26] Almost all listed components have anti-inflammatory effects, including myrcene, \u03b2-caryophyllene, caryophyllene oxide, humulene, \u03b1-pinene, linalool, limonene, terpinolene, \u03b3-terpinene, nerolidol, borneol, fenchone, and \u03b2-eudesmol. Some components have positive impacts on cancer and tumor treatments, such as \u03b2-caryophyllene, which also has a synergistic anticancer effect with humulene. Iso-caryophyllene, humulene, \u03b1-pinene, and linalool have antitumor activity.[27] All phenotypes of CS and TS had high proportions of caryophyllene oxide, which has a positive effect on type 1 and type 2 diabetes, cardiovascular diseases, and hypertension.[27][28] CII had the highest proportion of \u03b2-pinene, which has an important role in the regulation of diabetes, cancer, obesity, and other chronic diseases. Myrcene, terpinolene, linalool, and nerolidol have sedative effects. Humulene and limonene treat depression, and humulene has also been used in traditional medicine for treating insomnia, anxiety, delirium, and depression. With a significantly higher proportion of \u03b1-humulene, the CI phenotype could be used in further pharmaceutical research, especially if the total amount of essential oil could be increased in breeding procedures or affected by the environment. This phenotype also had a significantly higher proportion of \u03b2-eudesmol, which stimulates appetite.[27]\nThe total amount of detected monoterpenes was higher than that of sesquiterpenes in all three varieties.>[6][28] Comparing the investigated varieties, the highest monoterpene content (44.89%) occurred in TS, while CS varieties had the highest amount of sesquiterpenes (28.67%). In the group of monoterpenes belong compounds \u03b1-pinene, camphene, \u03b2-pinene, myrcene, 3-carene, \u03b1-terpinene, p-cimene, limonene, \u03b3-terpinene, fenchone, terpinolene, linalool, camphor, isoborneol, borneol, menthol, \u03b1-terpineol, \u03b2-citronellol, neryl acetate, and geranyl acetate, while in the group of sesquiterpenes belong compounds \u03b1-cedrene, \u03b2-caryophyllene, \u03b1-humulene, cis-nerolidol, geranyl isobutyrate, caryophyllene oxide, \u03b2-eudesmol, \u03b1-bisabolol, and phytol.\nA difference between varieties was observed when the average of essential oil components in individual phenotypes was compared (Figure 1). Principal component 1 (PC1) explained 39.44%, and PC2 explained 25.90% of the variance. Phenotypes of FS and TS were clustered together, while phenotypes of CS were separated. Considering the average amount of compounds in essential oil phenotypes within FS and TS, they were more uniform than CS phenotypes. (More detailed PCA plots with all analyzed variables are presented in Figure S1, Supplementary materials.)\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 Principal component analysis (PCA) plots for phenotype averages according to essential oil components made on the first two PC scores (PC1 explained 39.44%, and PC2 explained 25.90%), with a total variance of 65.34%.\n\n\n\nAccording to all analyzed components of essential oil, each individual variety of CS, TS, and FS was analyzed separately. Significant differences between both phenotypes of CS that were found by ANOVA can clearly also be observed in the PCA (Figure 2). TS and FS phenotypes were not distinguishable, which means that these two varieties were more uniform than CS.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2 Principal component analysis (PCA) plots for phenotypes of the Carmagnola Selected variety, according to components of essential oil made on the first two PC scores (PC1 explained 42.63%, and PC2 explained 16.16%), with a total variance of 58.79%.\n\n\n\nChemical analysis of cannabinoids in hemp (Cannabis sativa L.) phenotypes \nRegarding the analysis of cannabinoids, 10 different cannabinoids were identified in the 11 included phenotypes using high-performance liquid chromatography (HPLC) (Table 3). There was a significant difference in CBDA and CBGA cannabinoids between the CI and CII phenotypes. CII was more related to TS phenotypes. There were also significant differences in cannabinoids CBD and CBC between FS phenotypes and all other phenotypes.\n\n\n\n\n\n\n\nTable 3. Average cannabinoid content (%) of Carmagnola Selected, Tiborszallasi, and Finola Selection. Groups (a, b, c, and d) were formed by analysis of variance (ANOVA) using different hemp phenotypes. The same letters present similarities between phenotypes, while different letters present differences between phenotypes. Values are presented as the mean \u00b1 standard deviation (SD).\n\n\nCannabinoid\n\nCI\n\nSD\n\nCII\n\nSD\n\nTI\n\nSD\n\nTII\n\nSD\n\nTIII\n\nSD\n\nTIV\n\nSD\n\nTV\n\nSD\n\nFI\n\nSD\n\nFII\n\nSD\n\nFIII\n\nSD\n\nFIV\n\nSD\n\n\nCBD\n\n0.05\r\na\n\n0.04\n\n0.10\r\na\n\n0.04\n\n0.07\r\na\n\n0.03\n\n0.04\r\na\n\n0.02\n\n0.05\r\na\n\n0.02\n\n0.07\r\na\n\n0.04\n\n0.06\r\na\n\n0.03\n\n0.60\r\nb\n\n0.08\n\n0.78\r\nc\n\n0.12\n\n0.61\r\nb\n\n0.12\n\n0.64\r\nb\n\n0.19\n\n\nCBDA\n\n1.70\r\na\n\n1.53\n\n3.78\r\nb\n\n0.94\n\n3.22\r\nb\n\n1.11\n\n3.11\r\nb\n\n0.56\n\n2.65\r\na,b\n\n1.06\n\n3.78\r\nb\n\n0.62\n\n2.93\r\nb\n\n1.37\n\n6.48\r\nc\n\n0.35\n\n6.41\r\nc\n\n0.35\n\n6.36\r\nc\n\n0.25\n\n6.59\r\nc\n\n0.24\n\n\nCBG\n\n0.04\r\nb,c\n\n0.02\n\n0.04\r\nb,c\n\n0.01\n\n0.01\r\na\n\n0.01\n\n0.01\r\na\n\n0.01\n\n0.01\r\na\n\n0.01\n\n0.02\r\na,b\n\n0.02\n\n0.02\r\na\n\n0.01\n\n0.05\r\nc,d\n\n0.01\n\n0.05\r\nc,d\n\n0.01\n\n0.06\r\nd\n\n0.01\n\n0.06\r\nd\n\n0.02\n\n\nCBGA\n\n1.62\r\nb\n\n2.39\n\n0.44\r\na\n\n0.24\n\n0.25\r\na\n\n0.08\n\n0.16\r\na\n\n0.05\n\n0.19\r\na\n\n0.06\n\n0.22\r\na\n\n0.07\n\n0.18\r\na\n\n0.07\n\n1.00\r\na,b\n\n0.37\n\n0.63\r\na,b\n\n0.16\n\n0.91\r\na,b\n\n0.23\n\n1.11\r\na,b\n\n0.26\n\n\n\u03949-THC\n\n0.05\r\na,b,c\n\n0.09\n\n0.04\r\na,b,c\n\n0.05\n\n0.04\r\na,b,c\n\n0.03\n\n0.01\r\na\n\n0.01\n\n0.02\r\na,b\n\n0.02\n\n0.01\r\na\n\n0.01\n\n0.06\r\nb,c,d\n\n0.05\n\n0.08\r\nc,d\n\n0.02\n\n0.11\r\nd\n\n0.02\n\n0.08\r\nc,d\n\n0.01\n\n0.08\r\nc,d\n\n0.02\n\n\n\u03949-THCA\n\n0.91\r\na,b\n\n1.81\n\n0.75\r\na,b\n\n1.26\n\n0.71\r\na,b\n\n0.72\n\n0.14\r\na\n\n0.03\n\n0.53\r\na,b\n\n0.54\n\n0.18\r\na\n\n0.04\n\n1.39\r\nb\n\n1.23\n\n0.50\r\na,b\n\n0.08\n\n0.46\r\na,b\n\n0.07\n\n0.47\r\na,b\n\n0.06\n\n0.51\r\na,b\n\n0.05\n\n\nCBC\n\n0.01\r\na,b\n\n0.00\n\n0.02\r\na\n\n0.01\n\n0.01\r\na,b\n\n0.01\n\n0.00\r\na\n\n0.00\n\n0.01\r\na,b\n\n0.01\n\n0.01\r\na,b\n\n0.01\n\n0.01\r\na,b\n\n0.00\n\n0.05\r\nc\n\n0.01\n\n0.06\r\nd\n\n0.01\n\n0.05\r\nc\n\n0.01\n\n0.05\r\nc\n\n0.01\n\n\nCBCA\n\n0.20\r\na\n\n0.06\n\n0.27\r\na\n\n0.09\n\n0.34\r\na\n\n0.21\n\n0.24\r\na\n\n0.10\n\n0.25\r\na\n\n0.07\n\n0.34\r\na\n\n0.16\n\n0.30\r\na\n\n0.16\n\n0.62\r\nb\n\n0.10\n\n0.59\r\nb\n\n0.09\n\n0.64\r\nb\n\n0.10\n\n0.64\r\nb\n\n0.10\n\n\n\nCBDA represented the highest proportion of cannabinoids in all included phenotypes, with the highest content in the four FS phenotypes (6.36\u20136.59%). Compared to the other phenotypes, the highest proportion of CBGA was in phenotype CI (1.62%), followed by all FS phenotypes. Phenotype TV and CI had the highest proportion of \u03949-THCA, while the highest proportions of \u03949-THC were in the phenotypes of FS (0.08\u20130.11%). The lowest proportion of \u03949-THCA and \u03949-THC were in phenotypes TII and TIV. The CBN values were less than 0.04%, and \u03948-THC could not be detected in any sample. \nIn prior research, Aizpurua-Olaizola et al.[29] determined the highest proportion of \u03949-THCA, CBDA, and CBGA in clones of mother plants from chemotypes I, II, and III in an unknown variety. Additionally, Glivar et al.[30] analyzed the cannabinoid content in 15 different hemp varieties; when comparing proportions of CBDA, CBD, THC, and \u03949-THCA in bracts of TS, their values were higher than those found in our samples, while other proportions were comparable with our data. In CS, there were higher proportions of all components, compared to the components in our CS variety, except for \u03949-THCA, which was higher in our CS samples.[30]\nAs mentioned before, Small and Beckstead[11] and Fournier et al.[12] have already determined five major chemotypes based on the chemical profile of different hemp plants, whether it is marijuana or hemp type; however, our classification further divides varieties in different phenotypes. Nevertheless, based on almost the same ratio between CBD and \u03949-THC and CBDA and \u03949-THCA, phenotypes CI and TV belong to chemotype II, while all other phenotypes were classified as chemotype III due to a high CBDA and low \u03949-THCA content. None of the phenotypes were chemotype I, which was expected since the investigation included hemp and not marijuana varieties.\nPCA plots for the averages of included phenotypes in all three varieties were performed according to cannabinoid content. PC1 explained 67.57%, and PC2 explained 15.70% of the variance seen in cannabinoid content. (More detailed PCA plots with all analyzed variables are presented in Figure S2, Supplementary materials.) There was a considerable difference between the phenotypes of FS compared to CS and TS, but we could not completely differentiate between varieties CS and TS. Nevertheless, there were greater differences between TS and CS phenotypes than between phenotypes within FS. The results are presented in Figure 3. Greater differences were observed based on essential oil proportions than proportions of cannabinoid content.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 3 Principal component analysis (PCA) plot averages of analyzed cannabinoids for included phenotypes made on the first two PC scores (PC1 explained 67.57%, and PC2 explained 14.70%), with a total variance of 82.27%.\n\n\n\nEach variety was analyzed separately to identify any differences between phenotypes within varieties. As with the essential oil components, differences in cannabinoids between CS phenotypes (CI and CII) were detected, while there were no differences between the five TS and four FS phenotypes. An additional PCA was performed for the CS variety, and clear differences between the two phenotypes were observed (Figure 4). PC1 explained 41.89%, and PC2 explained 22.97% of the variance in cannabinoid content.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 4 Principal component analysis (PCA) plots of cannabinoid content for the Carmagnola Selected variety without \u03948-THC, made on the first two PC scores (PC1 explained 41.89%, and PC2 explained 22.97%), with a total variance of 64.86%.\n\n\n\nAizpurua-Olaizola et al.[29] discovered differences between chemotypes and correlations with certain terpenoids and cannabinoids in different chemotypes. In that research, clones of different chemotypes were used. They discovered a higher similarity between chemotypes II and III than chemotype I. In our research, there were no evident differences between these two chemotypes. Terpenoids such as \u03b3-selinene, \u03b2-selinene, \u03b1-gurgujene, \u03b3-elemene, selina-3,7(11)-diene, and \u03b2-curcumene had a higher correlation with THCA and a negative correlation with CBDA in chemotype I, while terpenoids like \u03b2-eudesmol, \u03b3-eudesmol, guaiol, \u03b1-bisabolene, \u03b1-bisabolol, and eucalyptol had a higher correlation with CBDA and negative correlation with THCA in chemotype III.[29] However, Hillig[31] reported that the components \u03b2-eudesmol, \u03b3-eudesmol, and guaiol were characteristic of C. indica, which has a higher level of \u03949-THC. In our case, the CI phenotype had the highest relative level of \u03b2-eudesmol and a high level of \u03949-THCA. When comparing phenotypes of each variety separately, only CS phenotypes showed significant differences in CBDA and CBGA, while phenotypes of TS and FS were not distinguishable, except FI and FIV, where separation from other phenotypes within the variety was evident in PCA analysis. This also indicated that TS and FS varieties were more uniform than CS, although CS is a variety selected from Carmagnola, which is the oldest hemp landrace grown in Italy and has been used as a breeding parent for many new varieties.[18]\n\nChemical analysis of both cannabinoids and essential oil in hemp (Cannabis sativa L.) phenotypes \nDue to the positive effects of C. sativa on human health, it is slowly gaining acceptance as a medicine. Due to the human endocannabinoid system, cannabinoids bind to different receptors through which they act on different diseases.[7] Numerous research studies have been conducted on the medicinal use of the Cannabis plant. Ware et al.[32] described 30 symptoms and diseases for which cannabis is used and most commonly reported. The most studied are multiple sclerosis, neuropathy, chronic pain, depression, arthritis, migraine, allergy, spinal pain asthma, and weight loss. Very important clinical studies were conducted also on treatments of several cancers, Alzheimer\u2019s disease, diabetes, epilepsy, Huntington\u2019s diseases, Tourette\u2019s syndrome, and also for treating HIV\/AIDS.[29] The benefits of using industrial hemp as medicine are in the low content of THC (lower than 0.3\u20131%), having no value as a recreational drug. Hemp with high CBD or other cannabinoid content, together with low THC content, could be available for general use without medical prescription and, on the other hand, farmers could grow industrial hemp for medicinal purposes without any special restrictions due to THC legislation. Consumption of hemp seeds has also positive effects on human health, especially because of the content of omega-3 and omega-6 fatty acids. Seeds contain also proteins, carbohydrates, oil and fibers, and minerals.[33]\nAccording to our results, specific phenotypes can be used for further breeding purposes with specific cannabinoid and\/or terpenoid content, which could be used as supplementary ingredients or in pharmacy. Lewis et al.[34] reported different chemovars of Cannabis with desirable cannabinoid and terpenoid profiles. Comparing our results with their research, the CII and TIV phenotypes had a high proportion of \u03b1-pinene and a high proportion of CBDA, which could be of interest due to their ability to improve learning and memory and as a modulator of THC overdose. TIV also had the most favorable ratio between total THC:CBD and high CBD content and \u03949-THC under 0.2%, which is the limit for varieties registered on the E.U. variety list. This phenotype could be very interesting in the pharmaceutical industry for the production of CBD. Due to its low THC content, it could also be grown outdoors by farmers. Phenotype TV had a high proportion of \u03b1-pinene, a relatively high proportion of myrcene, and a THCA and CBDA ratio around 1:2. Myrcene is responsible for the sedative effects of cannabis, and together with THC, it causes the \u201ccouch-lock\u201d effect. All FS phenotypes had the highest amount of essential oil and cannabinoids, with especially high CBD, limonene, \u03b2-caryophyllene, and \u03b1-humulene proportions, which relieves pain and inflammation and treats addiction.[34] In the past, cannabis inflorescences have been used in traditional medicine for the treatment of various diseases, such as acute pain, wound healing, insomnia, cough, and mania, which are also the subject of modern medicine studies related to cannabinoids.[35]\nFigure 5 explains Pearson\u2019s correlation coefficient of all three varieties and their phenotypes based on the main terpenoids and cannabinoids. Only a correlation coefficient higher than 0.7 was considered a significant correlation. As seen in Figure 5, \u03b1-pinene and \u03b2-pinene had a strong positive correlation. The cannabinoid CBDA had a strong positive correlation with the sesquiterpene \u03b1-bisabolol and cannabinoids CBD, CBC, and CBCA. Additionally, CBG showed a strong positive correlation with CBD and CBC. CBD had a negative correlation with caryophyllene oxide. Conversely, there was a positive correlation between CBG and cannabinoids CBD, CBC, and CBCA. CBC had a strong positive correlation with \u03b1-bisabolol and CBCA.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 5 Pearson\u2019s correlation coefficient of all three varieties and all phenotypes based on main terpenoids and cannabinoids.\n\n\n\nOur research was based on outdoor hemp cultivation. Industrial hemp is typically cultivated outside, where growing conditions are different from indoor cultivation. Therefore, the results of outdoor cultivation can vary from those of hemp cultivated indoors, where plants are exposed to more uniform conditions. In indoor cultivation, growing temperatures, day length, light, fertilization, irrigation, pest control, and harvest time are tightly determined to be optimal for growth.[36] Plants cultivated indoors are more stable and suitable for reproduction than plants grown outdoors.[16] However, our purpose was to identify the most suitable phenotypes that could be grown outdoors with appropriate technology and contain a high cannabinoid and terpenoid content.[36]\n\nMaterials and methods \nPlant material \nThree different dioecious hemp (Cannabis sativa L.) varieties (Cannabaceae) were grown in 2019: Carmagnola Selected (CS), Tiborszallasi (TS), and Finola Selection (FS). Two different phenotypes of CS (CI, CII), five phenotypes of TS (TI, TII, TIII, TIV, TV), and four phenotypes of FS (FI, FII, FIII, FIV) were selected according to visual traits (habitus, leaf size, length and compactness of the inflorescence, color of leaf petioles, and internode length) (Table 1). Plants were observed one to two times weekly throughout the whole growing season, and final phenotypes were determined at the end of maturity. The inflorescences of 10 plants of each phenotype were separately sampled according to their aforementioned morphological characteristics at their optimal maturity time. For chemical analysis, the inflorescences of five plants per phenotype were used. Plants began flowering in the middle of August and were sampled and harvested at the end of September when most of the pistils turned brown. Male plants were removed from the crop before flowering to prevent female pollination and seed development. The lowest average temperatures were in the middle of May (11.4 \u00b0C) and the highest temperatures were in June (22.7 \u00b0C). Temperatures were higher than usual except in May when temperatures were lower than usual. From June to September, there was more precipitation than in 30 years on average. The soil was fertilized with organic fertilizer before sowing. No irrigation and no pest control were used. Weeds were controlled only mechanically twice at the beginning of growing.\n\nStandards and reagents \nThe following terpenoid standards were purchased from Supelco: \u03b1-pinene, camphene, \u03b2-pinene, myrcene, 3-carene, \u03b1-terpinene, p-cymene, limonene, \u03b3-terpinene, fenchone, terpinolene, linalool, camphor, isoborneol, borneol, menthol, \u03b1-terpineol, \u03b2-citronellol, neryl acetate, geranyl acetate, \u03b1-cedrene, \u03b2-caryophyllene, \u03b1-humulene, cis-nerolidol, geranyl isobutyrate, caryophyllene oxide, \u03b2-eudesmol, \u03b1-bisabolol, and phytol. Cannabinoid standards were purchased from Sigma Aldrich (1 mg\/mL): cannabidiolic acid (CBDA), cannabigerolic acid (CBGA), cannabigerol (CBG), cannabidiol (CBD), cannabinol (CBN), \u03949-tetrahydrocannabinol (\u03949-THC), \u03948-tetrahydrocannabinol (\u03948-THC), cannabichromene (CBC), \u03949-tetrahydrocannabinolic acid (\u03949-THCA), and cannabichromenic acid (CBCA). Acid standards were dissolved in acetonitrile, and nonacid standards were dissolved in methanol. Hexane, ammonium format, and acetonitrile were purchased from Sigma Aldrich, methanol was purchased from J. T. Baker, and formic acid was purchased from Scharlau. Chemical structures of essential oils and cannabinoids were referenced from various sources[37][38][39] and are presented in the Supplementary materials.\n\nSample preparation \nHarvested inflorescences without stems and the larger leaves were dried in the drier at 30 \u00b0C immediately after sample collection. Inflorescences were ground (30\u201350 g) and stored in the refrigerator at \u221220 \u00b0C until analysis. For gas chromatography (GC) with flame ionization detection (GC-FID), dried samples (27\u201330 g) of the 11 different phenotypes were steam distilled (3 hours, 1 L distilled water) by a Clevenger apparatus. Each plant within each phenotype was analyzed separately with five different plants per phenotype. The collected oil was diluted with 1 mL of hexane and separated with gas chromatography analysis. For HPLC analysis, ground flowers (200 \u00b1 9 mg) of 11 different phenotypes were weighed in 50 mL centrifuge tubes, and 25.0 mL of 80% methanol was added before vortexing for 30 seconds. Extraction was performed by sonicating in a bath for 15 minutes and vortexed every five minutes.[40]\n\nGC\u2013FID analysis \nAn Agilent GC 6890 (Agilent) with an HP 1 capillary column (Crosslinkeed Methyl Silicone Gum) of 2.5 mm \u00d7 0.2 mm \u00d7 0.1 mm (Agilent, Santa Clara, CA, USA) was used. The carrier gas flow (Helium 5.0) was set to 0.5 mL\/min. One \u00b5L of the solution was injected into the injector port at a temperature of 180 \u00b0C. The oven temperature program was 3.5 minutes at 60 \u00b0C, 3.5 \u00b0C\/minute to 155 \u00b0C, and 30 \u00b0C\/minute to 300 \u00b0C. Detection was carried out on a flame ionization detector set at 300 \u00b0C with a 42.47 min runtime. Identification was made with ChemStation software for GC.\n\nHPLC analysis \nFor HPLC, an Agilent 1200 (Agilent, Santa Clara, CA, USA) was used. Separation was achieved on a Raptor ARC-18 (octadecylsilane) with a 2.7 \u00b5m, 150 mm \u00d7 4.6 mm ID (Restek, Bellefonte, PA, USA) column. Mobile phase A was composed of 5 mM ammonium formate and 0.1% formic acid in deionized water with a final pH of 3.1. Mobile phase B was composed of acetonitrile and 0.1% formic acid. 25% mobile phase A, and 75% mobile phase B. The flow rate was 1.5 mL\/minute, and an isocratic elution was in use. A calibration curve for each standard ranging from 5 to 250 \u00b5g\/mL was prepared with correlation coefficients higher than 0.9975. The extracts were filtered through a disposable syringe with a PTFE filter (0.45 \u00b5m). The injection volume was 5 \u00b5L. Detection took place at 228 nm with a 15 minute run time. Identification was carried out with ChemStation for LC.\n\nStatistical analyses \nPrincipal component analysis was performed using OriginPro 2021. Data were plotted based on the first two components. Multifactor ANOVA with Duncan test at a 95% confidential interval was performed in Statgraphics Centurion Software. Eleven phenotypes were compared according to the components of essential oil and cannabinoids, which were selected according to morphological differences between phenotypes. Means and standard deviations were calculated in Microsoft Excel.\n\nConclusions \nThe purpose of our study was to compare the composition of essential oil and cannabinoids in 11 hemp phenotypes from three varieties based on morphological distinctness. To our knowledge, this is the first research to study differences in cannabinoid and essential oil content in various phenotypes within different varieties of hemp. Our study showed that based on the content of cannabinoids and essential oil, FS significantly and positively differed from CS and TS varieties, making it important for the hemp industry. Based on PCA analysis, there were no evident differences between phenotypes in FS and TS within each analyzed individual variety, while the two CS phenotypes were distinctive. By comparing the average content of the detected compounds in essential oil and cannabinoids between individual phenotypes of all three varieties, differences were determined. Among all analyzed phenotypes, CI, TII, TIV, and TV showed an interesting ratio between THC and CBD for further use in the pharmaceutical industry and breeding. CI and TV had a similar ratio of THC and CBD, while TII and TIV had a very high level of CBD, which may be useful in the pharmaceutical industry for epilepsy treatment.[41] Additionally, the phenotypes of FS could be interesting due to a high CBD content, which could be extracted. When comparing phenotypes, according to ANOVA, significant differences in individual components of essential oil and cannabinoids were shown, potentiating the use of individual phenotypes in specialized breeding programs or disease treatments. Cannabinoids and terpenoids may have synergistic effects on different disease symptoms. When breeding or growing selected phenotypes, the plant varieties will be more homogenous regarding their chemical composition and, consequently, more interesting for pharmaceutical use. The cultivation of C. sativa with a low content of \u03949-THC gives farmers an opportunity to cultivate hemp outdoor under controlled conditions for pharmaceutical purposes.\n\nSupplementary materials \nThe following files are held within this downloadable .zip file:\n\n Figure S1: Principal component analysis (PCA) plot for three different hemp populations (Carmagnola Selected, Tiborszallasi, and Finola Selection) according to 29 components of essential oil, made on first two PC scores (PC1 explained 24.43%, and PC2 explained 17.53%), with a total variance of 41.96%\n Figure S2: PCA plot for all three populations (Carmagnola Selected, Tiborszallasi, and Finola Selection) according to all cannabinoids made on first two PC scores (PC1 explained 50.71%, and PC2 explained 17.00%) with a total variance of 67.71%\n Figure S3: PCA plot for all analyzed samples of all populations according to components of essential oil and cannabinoids made on the first two PC scores (PC1 explained 27.38%, and PC2 explained 16.20%) with a total variance of 43.58%\n Figure S4: Phenotype CI: Plants of CI were light green with large primary leaves\n Figure S5: Phenotype CII: Plants of the CII phenotype were dark green with narrow leaflets and anthocyanin coloration on the leaf petiole. Most plants had more lateral branches\n Figure S6: Phenotype TI: TI plants were taller compared to the other phenotypes with lateral branches and smaller inflorescences\n Figure S7: Phenotype TII: TII plants were dark green with anthocyanin coloration of leaf petioles on the primary and secondary branches \n Figure S8: Phenotype TIII: TIII plants were also dark green, smaller than other phenotypes, with less lateral branches, anthocyanin coloration of the leaf petioles, and pink bracts\n Figure S9: Phenotype TIV: TIV plants were dark green with big compact inflorescences and thin leaflets around the inflorescences\n Figure S10: TV plants were small and had strong anthocyanin coloration of upper and lower leaves, inflorescences, and leaf petioles\n Figure S11: Phenotype FI: Plants of the FI phenotype were dark green with compact growth and taller than other phenotypes of FS. Plants had long side branches with longer and larger leaves and higher main inflorescences\n Figure S12: Phenotype FII: FII plants were medium green with shorter side branches and compact inflorescences surrounded by many leaflets\n Figure S13: Phenotype FIII: Plants from the FIII phenotype were light green with long side branches and a wide angle between them. Leaves had medium anthocyanin-colored edges, and inflorescences had green leaflets and were not compact\n Figure S14: Phenotype FIV: FIV plants were dark green with long inflorescences, and long, upright side branches. Leaf petioles had strong anthocyanin coloration\n Table S1: Average essential oil (EO) content (mL\/100 g of air-dried hemp) in the inflorescences, average composition (%) of essential oil, and groups (a, b, c, and d) from statistical analysis performed by analysis of variance (ANOVA) from different phenotypes of Carmagnola Selected, Tiborszallasi, and Finola Selection. The same letters present similarities between phenotypes, while different letters present differences between phenotypes according to components of essential oil. The mean \u00b1 standard deviation (SD) is reported\n Table S2: Chemical structures of components of essential oil and cannabinoids described in the research\nAcknowdledgements \nAuthor contributions \nConceptualization, A.\u010c., S.K. and M.E.; data curation, M.E.; formal analysis, M.E.; investigation, M.E.; methodology, I.J.K., M.O. and M.E.; resources, M.E. and A.\u010c.; supervision, A.\u010c. and I.J.K.; visualization, M.E. and A.\u010c.; writing\u2014original draft preparation, M.E.; writing\u2014review and editing, A.\u010c., I.J.K., M.O. and S.K. All authors have read and agreed to the published version of the manuscript.\n\nFunding \nThis research was funded by the Slovenian Research Agency (ARRS).\n\nConflicts of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 Hillig, K.W. (2005). \"Genetic evidence for speciation in Cannabis (Cannabaceae)\". Genetic Resources and Crop Evolution 52: 161\u201380. doi:10.1007\/s10722-003-4452-y.   \n\n\u2191 Crini, G.; Lichtfouse, E.; Chanet, G. et al. (2020). \"Applications of hemp in textiles, paper industry, insulation and building materials, horticulture, animal nutrition, food and beverages, nutraceuticals, cosmetics and hygiene, medicine, agrochemistry, energy production and environment: A review\". Environmental Chemistry Letters 18: 1451\u201376. doi:10.1007\/s10311-020-01029-2.   \n\n\u2191 3.0 3.1 3.2 Namdar, D.; Mazuz, M.; Ion, A. et al. (2018). \"Variation in the compositions of cannabinoid and terpenoids in Cannabis sativa derived from inflorescence position along the stem and extraction methods\". Industrial Crops and Products 113: 376\u201382. doi:10.1016\/j.indcrop.2018.01.060.   \n\n\u2191 Turner, J.C.; Hemphill, J.K.; Mahlberg, P.G. (1978). \"Quantitative Determination of Cannabinoids in Individual Glandular Trichomes of Cannabis sativa L. (Cannabaceae)\". American Journal of Botany 65 (10): 1103\u20136. doi:10.1002\/j.1537-2197.1978.tb06177.x.   \n\n\u2191 Hanu\u0161, L.O.; Meyer, S.M.; Mu\u00f1oz, E. et al. (2016). \"Phytocannabinoids: A unified critical inventory\". Natural Product Reports 33: 1357\u201392. doi:10.1039\/C6NP00074F.   \n\n\u2191 6.0 6.1 6.2 6.3 Bertoli, A.; Tozzi, S.; Pistelli, L. et al. (2010). \"Fibre hemp inflorescences: From crop-residues to essential oil production\". Industrial Crops and Products 32 (3): 329\u201337. doi:10.1016\/j.indcrop.2010.05.012.   \n\n\u2191 7.0 7.1 7.2 Andre, C.M.; Hausman, J.-F.; Guerriero, G. (2016). \"Cannabis sativa: The plant of the thousand and one molecules\". Frontiers in Plant Medicine 7: 19. doi:10.3389\/fpls.2016.00019. PMC PMC4740396. PMID 26870049. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396 .   \n\n\u2191 Russo, E.B. (2011). \"Taming THC: potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\". British Journal of Pharmacology 163 (7): 1344\u201364. doi:10.1111\/j.1476-5381.2011.01238.x.   \n\n\u2191 da Silva, A.C.R.; Lopes, P.M.; de Azevedo, M.M.B. et al. (2012). \"Biological Activities of a-Pinene and \u03b2-Pinene Enantiomers\". Molecules 17 (6): 6305-6316. doi:10.3390\/molecules17066305.   \n\n\u2191 Appendino, G.; Gibbons, S.; Giana, A. et al. (2008). \"Antibacterial Cannabinoids from Cannabis sativa: A Structure\u2212Activity Study\". Journal of Natural Products 71 (8): 1427\u201330. doi:10.1021\/np8002673.   \n\n\u2191 11.0 11.1 Small, E.; Beckstead, H.D. (1973). \"Common cannabinoid phenotypes in 350 stocks of Cannabis\". Lloydia 36 (2): 144\u201365. PMID 4744553.   \n\n\u2191 12.0 12.1 Fournier, G.; Richez-Dumanois, C.; Duvezin, J. et al. (1987). \"Identification of a new chemotype in Cannabis sativa: cannabigerol-dominant plants, biogenetic and agronomic prospects\". Planta Medica 53 (3): 277\u201380. doi:10.1055\/s-2006-962705. PMID 3628560.   \n\n\u2191 Pollio, A. (2016). \"The Name of Cannabis: A Short Guide for Nonbotanists\". Cannabis and Cannabinoid Research 1 (1): 234-238. doi:10.1089\/can.2016.0027. PMC PMC5531363. PMID 28861494. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5531363 .   \n\n\u2191 Schilling, S.; Melzer, R.; McCabe, P.F. (2020). \"Cannabis sativa\". Current Biology 30 (1): R8\u2013R9. doi:10.1016\/j.cub.2019.10.039. PMID 31910378.   \n\n\u2191 15.0 15.1 Sawler, J.; Stout, J.M.; Gardner, K.M. et al. (2015). \"The Genetic Structure of Marijuana and Hemp\". PLoS One 10 (8): e0133292. doi:10.1371\/journal.pone.0133292. PMC PMC4550350. PMID 26308334. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350 .   \n\n\u2191 16.0 16.1 Janatov\u00e1, A.; Fra\u0148kov\u00e1, A.; Tlusto\u0161, P. et al. (2018). \"Yield and cannabinoids contents in different cannabis (Cannabis sativa L.) genotypes for medical use\". Industrial Crops and Products 112: 363\u201367. doi:10.1016\/j.indcrop.2017.12.006.   \n\n\u2191 Aguilar, S.; Guti\u00e9rrez, V.; S\u00e1nchez, L. et al. (23 April 2018). \"Medicinal cannabis policies and practices around the world\". International Drug Policy Consortium. https:\/\/idpc.net\/publications\/2018\/04\/medicinal-cannabis-policies-and-practices-around-the-world . Retrieved 19 April 2021 .   \n\n\u2191 18.0 18.1 Ranalli, P. (2004). \"Current status and future scenarios of hemp breeding\". Euphytica 140: 121\u201331. doi:10.1007\/s10681-004-4760-0.   \n\n\u2191 \"Cerealicoltura e Colture Industriali\". Consiglio per la ricerca in agricoltura e l'analisi dell'economia agraria. https:\/\/www.crea.gov.it\/web\/cerealicoltura-e-colture-industriali . Retrieved 20 April 2021 .   \n\n\u2191 \"National Food Chain Safety Office (NEBIH)\". ERA-LEARN. European Union. https:\/\/www.era-learn.eu\/network-information\/organisations\/national-food-chain-safety-office-1 . Retrieved 20 April 2021 .   \n\n\u2191 \"Finnish Food Authority\". Finnish Food Authority. https:\/\/www.ruokavirasto.fi\/en\/ . Retrieved 20 April 2021 .   \n\n\u2191 Small, E.; Antle, T. (2003). \"A Preliminary Study of Pollen Dispersal in Cannabis sativa in Relation to Wind Direction\". Journal of Industrial Hemp 8 (2): 37-50. doi:10.1300\/J237v08n02_03.   \n\n\u2191 23.0 23.1 Nissen, L.; Zatta, A.; Stefanini, I. et al. (2010). \"Characterization and antimicrobial activity of essential oils of industrial hemp varieties (Cannabis sativa L.)\". Fitoterapia 81 (5): 413\u201319. doi:10.1016\/j.fitote.2009.11.010.   \n\n\u2191 Novak, J.; Zitteril-Eglseer, K.; Deans, S.G. et al. (2001). \"Essential oils of different cultivars of Cannabis sativa L. and their antimicrobial activity\". Flavour and Fragrance Journal 16 (4): 259-262. doi:10.1002\/ffj.993.   \n\n\u2191 Novak, J.; Franz, C. (2003). \"Composition of the Essential Oils and Extracts of Two Populations of Cannabis sativa L. ssp. spontanea from Austria\". Journal of Essential Oil Research 15 (3): 158\u201360. doi:10.1080\/10412905.2003.9712100.   \n\n\u2191 Russo, E.B.; Marcu, J. (2017). \"Cannabis Pharmacology: The Usual Suspects and a Few Promising Leads\". Advances in Pharmacology 80: 67\u2013134. doi:10.1016\/bs.apha.2017.03.004. PMID 28826544.   \n\n\u2191 27.0 27.1 27.2 Nuutinen, T. (2018). \"Medicinal properties of terpenes found in Cannabis sativa and Humulus lupulus\". European Journal of Medicinal Chemistry 157: 198\u2013228. doi:10.1016\/j.ejmech.2018.07.076. PMID 30096653.   \n\n\u2191 28.0 28.1 Mudge, E.M.; Brown, P.N.; Murch, S.J. (2019). \"The Terroir of Cannabis: Terpene Metabolomics as a Tool to Understand Cannabis sativa Selections\". Planta Medica 85 (9\u201310): 781\u201396. doi:10.1055\/a-0915-2550. PMID 31096276.   \n\n\u2191 29.0 29.1 29.2 29.3 Aizpurua-Olaizola, O.; Soydaner, U.; \u00d6zt\u00fcrk, E. et al. (2016). \"Evolution of the Cannabinoid and Terpene Content during the Growth of Cannabis sativa Plants from Different Chemotypes\". Journal of Natural Products 79 (2): 324\u201331. doi:10.1021\/acs.jnatprod.5b00949. PMID 26836472.   \n\n\u2191 30.0 30.1 Glivar, T.; Er\u017een, J.; Kreft, S. et al. (2020). \"Cannabinoid content in industrial hemp (Cannabis sativa L.) varieties grown in Slovenia\". Industrial Crops and Products 145: 112082. doi:10.1016\/j.indcrop.2019.112082.   \n\n\u2191 Hillig, K.W. (2004). \"A chemotaxonomic analysis of terpenoid variation in Cannabis\". Biochemical Systematics and Ecology 32 (10): 875\u201391. doi:10.1016\/j.bse.2004.04.004.   \n\n\u2191 Ware, M.A.; Adams, H.; Guy, G.W. (2005). \"The medicinal use of cannabis in the UK: Results of a nationwide survey\". International Journal of Clinical Practice 59 (3): 291\u20135. doi:10.1111\/j.1742-1241.2004.00271.x. PMID 15857325.   \n\n\u2191 Callaway, J.C. (2004). \"Hempseed as a nutritional resource: An overview\". Euphytica 140: 65\u201372. doi:10.1007\/s10681-004-4811-6.   \n\n\u2191 34.0 34.1 Lewis, M.A.; Russo, E.B.; Smith, K.M. (2018). \"Pharmacological Foundations of Cannabis Chemovars\". Planta Medica 84 (4): 225\u201333. doi:10.1055\/s-0043-122240. PMID 29161743.   \n\n\u2191 Jin, D.; Dai, K.; Xie, Z. et al. (2020). \"Secondary Metabolites Profiled in Cannabis Inflorescences, Leaves, Stem Barks, and Roots for Medicinal Purposes\". Scientific Reports 10 (1): 3309. doi:10.1038\/s41598-020-60172-6. PMC PMC7039888. PMID 32094454. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7039888 .   \n\n\u2191 36.0 36.1 Chandra, S.; Lata, H.; ElSohly, M.A. et al. (2017). \"Cannabis cultivation: Methodological issues for obtaining medical-grade product\". Epilepsy & Behavior 70 (Pt B): 302\u201312. doi:10.1016\/j.yebeh.2016.11.029. PMID 28202406.   \n\n\u2191 \"MedChemExpress\". MedChemExpress. 19 April 2021. https:\/\/www.medchemexpress.com\/ .   \n\n\u2191 \"ChemSpider\". Royal Society of Chemistry. 19 April 2021. https:\/\/www.chemspider.com\/ .   \n\n\u2191 \"Chem\u00e9o\". C\u00e9ondo GmbH. 19 April 2021. https:\/\/www.chemeo.com\/ .   \n\n\u2191 Mudge, E.M.; Murch, S.J.; Brown, P.N. (2017). \"Leaner and greener analysis of cannabinoids\". Analytical and Bioanalytical Chemistry 409 (12): 3153\u201363. doi:10.1007\/s00216-017-0256-3.   \n\n\u2191 Neubauer, D.; Benedik, M.P.; Osredkar, D. (2018). \"Cannabidiol for treatment of refractory childhood epilepsies: Experience from a single tertiary epilepsy center in Slovenia\". Epilepsy & Behavior 81: 79\u201385. doi:10.1016\/j.yebeh.2018.02.009. PMID 29526578.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added.\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\">https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes<\/a>\n\t\t\t\t\tCategories: CannaQAwiki journal articles (added in 2021)CannaQAwiki journal articles (all)CannaQAwiki journal articles on cannabis researchCannaQAwiki journal articles on cannabis testing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tList of articles\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 15 June 2021, at 23:15.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout CannaQAWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t","d78732feb44acdd119e6c7a504f8afa4_html":"<body class=\"mediawiki ltr sitedir-ltr ns-170 ns-subject page-Journal_Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_Cannabis_sativa_L_phenotypes rootpage-Journal_Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_Cannabis_sativa_L_phenotypes skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Metabolomic analysis of cannabinoid and essential oil profiles in different hemp (<i>Cannabis sativa<\/i> L.) phenotypes<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Hemp\" title=\"Hemp\" class=\"wiki-link\" data-key=\"c23e30b6cf1df54f1dc338492c9f9da2\">Hemp<\/a> (<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_sativa\" title=\"Cannabis sativa\" class=\"wiki-link\" data-key=\"e003358742012354d1ff6002bc5781de\"><i>Cannabis sativa<\/i> L.<\/a>) <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabinoid\" title=\"Cannabinoid\" class=\"wiki-link\" data-key=\"c224c3041748677fcdce5b5209900b7b\">cannabinoids<\/a> and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Terpene\" title=\"Terpene\" class=\"wiki-link\" data-key=\"6bcbb95d582c08073101e9c4cfc91931\">terpenoids<\/a> have therapeutic effects on human and animal health. <i><a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis\" title=\"Cannabis\" class=\"wiki-link\" data-key=\"a70b76268930d795518ff1f98d7e500d\">Cannabis<\/a><\/i> plants can often have a relatively high heterogeneity, which leads to different <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Phenotype\" title=\"Phenotype\" class=\"wiki-link\" data-key=\"841d4796825869f94da28585ed94eabb\">phenotypes<\/a> that have different chemical profiles despite being from the same variety. Little information exists about cannabinoid and terpenoid profiles in different hemp phenotypes within the same variety. For this study, 11 phenotypes from three different varieties\u2014Carmagnola Selected (CS), Tiborszallasi (TS), and Finola Selection (FS)\u2014were analyzed. The components of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_concentrate\" title=\"Cannabis concentrate\" class=\"wiki-link\" data-key=\"4820fde4ed6a3627d7a670471f273d33\">essential oil<\/a> (29) were analyzed using <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Gas_chromatography\" title=\"Gas chromatography\" class=\"wiki-link\" data-key=\"5094f035cb8bead5003deb9181e398b8\">gas chromatography<\/a> with <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Chromatography_detector\" title=\"Chromatography detector\" class=\"wiki-link\" data-key=\"7787a159a1c7c0a3b2cc28bbc3189587\">flame ionization detection<\/a> (GC-FID), and 10 different cannabinoids of each phenotype were determined using <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=High-performance_liquid_chromatography\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"4c5d83ffa9785383c3eb0be7ea78dd2b\">high-performance liquid chromatography<\/a> (HPLC). \n<\/p><p>Principal component analysis (PCA) and analysis of variance (ANOVA) showed that according to the components of essential oil, FS and TS plants were more uniform than CS plants, where there were great differences between CI and CII phenotypes. The content of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabidiolic_acid\" title=\"Cannabidiolic acid\" class=\"wiki-link\" data-key=\"772927eeed478eff799c410179be18a1\">cannabidiolic acid<\/a> (CBDA) was the highest in all four FS phenotypes. By comparing cannabinoid profiles, FS was clearly separated from TS and CS, while these two varieties were not clearly distinguishable. Phenotypes TV and CI had the highest total content of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Tetrahydrocannabinol\" title=\"Tetrahydrocannabinol\" class=\"wiki-link\" data-key=\"15f3b3e338baeb54c04c715818759ec9\">tetrahydrocannabinol<\/a> (\u0394<sup>9<\/sup>-THC), while all phenotypes of FS had the highest total content of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabidiol\" title=\"Cannabidiol\" class=\"wiki-link\" data-key=\"bbfcdd40d5eda5d20d1d5ab368c922a0\">cannabidiol<\/a> (CBD). The highest total content of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabigerol\" title=\"Cannabigerol\" class=\"wiki-link\" data-key=\"6fc007b4c33a4c59a485b2c621035714\">cannabigerol<\/a> (CBG) was determined in phenotype CI. Obtained results are useful for the development of new supplementary ingredients, for different <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_(drug)\" title=\"Cannabis (drug)\" class=\"wiki-link\" data-key=\"6500790ffdd84957d4b055c47e48efca\">pharmacy treatments<\/a>, and for further breeding purposes.\n<\/p><p><b>Keywords<\/b>: <i>Cannabis sativa<\/i> L., Cannabaceae, cannabinoids, essential oils, terpenes, GC-FID, HPLC\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Hemp\" title=\"Hemp\" class=\"wiki-link\" data-key=\"c23e30b6cf1df54f1dc338492c9f9da2\">Hemp<\/a> (<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_sativa\" title=\"Cannabis sativa\" class=\"wiki-link\" data-key=\"e003358742012354d1ff6002bc5781de\"><i>Cannabis sativa<\/i> L.<\/a>) originated from central Asia and has been used for human and animal food, as a source of fiber for ropes, and in <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_(drug)\" title=\"Cannabis (drug)\" class=\"wiki-link\" data-key=\"6500790ffdd84957d4b055c47e48efca\">medicine<\/a>.<sup id=\"rdp-ebb-cite_ref-HilligGenetic05_1-0\" class=\"reference\"><a href=\"#cite_note-HilligGenetic05-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CriniAppli20_2-0\" class=\"reference\"><a href=\"#cite_note-CriniAppli20-2\">[2]<\/a><\/sup> It contains more than 500 <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Phytochemical\" title=\"Phytochemical\" class=\"wiki-link\" data-key=\"512240c65f3fdf05fdddb5d604c8838f\">phytochemicals<\/a> with many therapeutic purposes and has been used to treat epilepsy, Alzheimer\u2019s disease, Parkinson\u2019s disease, multiple sclerosis, pain and nausea in cancer patients, diabetes, and eating disorders.<sup id=\"rdp-ebb-cite_ref-NamdarVaria18_3-0\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-3\">[3]<\/a><\/sup>\n<\/p><p>The most well-known phytochemicals are secondary metabolites, such as <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabinoid\" title=\"Cannabinoid\" class=\"wiki-link\" data-key=\"c224c3041748677fcdce5b5209900b7b\">cannabinoids<\/a> and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Terpene\" title=\"Terpene\" class=\"wiki-link\" data-key=\"6bcbb95d582c08073101e9c4cfc91931\">terpenoids<\/a>.<sup id=\"rdp-ebb-cite_ref-TurnerQuant78_4-0\" class=\"reference\"><a href=\"#cite_note-TurnerQuant78-4\">[4]<\/a><\/sup> More than 150 cannabinoids have already been identified in hemp.<sup id=\"rdp-ebb-cite_ref-Hanu.C5.A1Phyto16_5-0\" class=\"reference\"><a href=\"#cite_note-Hanu.C5.A1Phyto16-5\">[5]<\/a><\/sup> The most active and studied compounds are <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Tetrahydrocannabinol\" title=\"Tetrahydrocannabinol\" class=\"wiki-link\" data-key=\"15f3b3e338baeb54c04c715818759ec9\">\u0394<sup>9<\/sup> tetrahydrocannabinol<\/a> (\u0394<sup>9<\/sup>-THC), <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabidiol\" title=\"Cannabidiol\" class=\"wiki-link\" data-key=\"bbfcdd40d5eda5d20d1d5ab368c922a0\">cannabidiol<\/a> (CBD), <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabigerol\" title=\"Cannabigerol\" class=\"wiki-link\" data-key=\"6fc007b4c33a4c59a485b2c621035714\">cannabigerol<\/a> (CBG), <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabichromene\" title=\"Cannabichromene\" class=\"wiki-link\" data-key=\"1fa141abe89973bfa85a54e2632edf31\">cannabichromene<\/a> (CBC), and their carboxylated forms.<sup id=\"rdp-ebb-cite_ref-NamdarVaria18_3-1\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-3\">[3]<\/a><\/sup> Terpenoids in <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_concentrate\" title=\"Cannabis concentrate\" class=\"wiki-link\" data-key=\"4820fde4ed6a3627d7a670471f273d33\">essential oil<\/a> are divided into monoterpenes and sesquiterpenes, which are responsible for hemp fragrance and flavor and also contribute to therapeutic effects. There are generally fewer sesquiterpenes than monoterpenes detected in hemp flowers. The highest content of cannabinoids and terpenoids is found in the glandular <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Trichome\" title=\"Trichome\" class=\"wiki-link\" data-key=\"40afd6e0542ec77b9084ef6d7bdb72bb\">trichomes<\/a> on bracts.<sup id=\"rdp-ebb-cite_ref-BertoliFibre10_6-0\" class=\"reference\"><a href=\"#cite_note-BertoliFibre10-6\">[6]<\/a><\/sup>\n<\/p><p>Precursors for cannabinoids have two biosynthetic pathways. The polyketide pathway leads to olivetolic acid (OLA), and the plastidial 2-C-methyl-D-erytritol 4-phosphate (MEP) pathway leads to geranyl diphosphate (GPP). Precursors OLA and GPP form <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabigerolic_acid\" title=\"Cannabigerolic acid\" class=\"wiki-link\" data-key=\"d0a5399af99137ebf5aa893d34531e49\">cannabigerolic acid<\/a> (CBGA), which is a precursor for different cannabinoids, as well as <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Tetrahydrocannabinolic_acid\" title=\"Tetrahydrocannabinolic acid\" class=\"wiki-link\" data-key=\"a047dcf0f7a85c57e1d2d308b56e5e4a\">tetrahydrocannabinolic acid<\/a> (THCA), <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabidiolic_acid\" title=\"Cannabidiolic acid\" class=\"wiki-link\" data-key=\"772927eeed478eff799c410179be18a1\">cannabidiolic acid<\/a> (CBDA), and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabichromenic_acid\" title=\"Cannabichromenic acid\" class=\"wiki-link\" data-key=\"e8e68540671d5adcbb1b2dac8d088b69\">cannabichromenic acid<\/a> (CBCA).<sup id=\"rdp-ebb-cite_ref-AndreCannabis16_7-0\" class=\"reference\"><a href=\"#cite_note-AndreCannabis16-7\">[7]<\/a><\/sup> Terpenoids are composed of isoprene units. Similar to cannabinoids, terpenoids also have different biosynthetic pathways. Sesquiterpenes and triterpenes are formed from the cytosolic mevalonic acid (MVA) pathway, while monoterpenes, diterpenes, and tetraterpenes are formed via the plastid-localized (MEP) pathway. Subsequently, precursors of sesquiterpene farnesyl diphosphate (FPP) and monoterpene geranyl diphosphate are formed.<sup id=\"rdp-ebb-cite_ref-AndreCannabis16_7-1\" class=\"reference\"><a href=\"#cite_note-AndreCannabis16-7\">[7]<\/a><\/sup>\n<\/p><p>Terpenoids and cannabinoids may have a synergistic effect on human and animal health.<sup id=\"rdp-ebb-cite_ref-RussoTaming11_8-0\" class=\"reference\"><a href=\"#cite_note-RussoTaming11-8\">[8]<\/a><\/sup> An example of the positive effects of the combined use of cannabinoids and terpenoids is acne therapy, in which CBD, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Limonene\" title=\"Limonene\" class=\"wiki-link\" data-key=\"5d6c7cd9e4515717a8c204576c778324\">limonene<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Linalool\" title=\"Linalool\" class=\"wiki-link\" data-key=\"065d30b67ffb79c4152ca0e04d58d40d\">linalool<\/a>, and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Pinene\" title=\"Pinene\" class=\"wiki-link\" data-key=\"6de2eb2cc03bfd4404a442d2f0e4076b\">pinene<\/a> are involved. Cannabinoids and terpenoids such as CBG and pinene also have a combined effect on MRSA (methicillin-resistant Staphylococcus aureus).<sup id=\"rdp-ebb-cite_ref-DaSilvaBiological12_9-0\" class=\"reference\"><a href=\"#cite_note-DaSilvaBiological12-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AppendinoAntibact08_10-0\" class=\"reference\"><a href=\"#cite_note-AppendinoAntibact08-10\">[10]<\/a><\/sup> However, the issue of synergy remains controversial and needs further investigation.\n<\/p><p>According to the chemical composition, there are five major hemp <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Chemotype\" title=\"Chemotype\" class=\"wiki-link\" data-key=\"2cee49cb83af61d36357ce99efa5ab72\">chemotypes<\/a>. Small and Beckstead<sup id=\"rdp-ebb-cite_ref-SmallCommon73_11-0\" class=\"reference\"><a href=\"#cite_note-SmallCommon73-11\">[11]<\/a><\/sup> determined three chemotypes: chemotype I, with a THC content higher than 0.3% and CBD content lower than 0.5%; chemotype II (intermediate type), with a THC and CBD ratio that is roughly equal; and chemotype III, with a higher CBD content than 0.5% and THC content lower than 0.3% of the flower dry matter. Later, Fournier <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-FournierIdent87_12-0\" class=\"reference\"><a href=\"#cite_note-FournierIdent87-12\">[12]<\/a><\/sup> determined two other chemotypes: chemotype IV, with a prevalence of CBG higher than 0.3% of the flower dry matter, and chemotype V, with an undetectable content of cannabinoids. \n<\/p><p>Numerous scientists have studied species and subspecies of <i>Cannabis<\/i>.<sup id=\"rdp-ebb-cite_ref-PollioTheName16_13-0\" class=\"reference\"><a href=\"#cite_note-PollioTheName16-13\">[13]<\/a><\/sup> In general, it is known that hemp and marijuana differ based on THC content. Hemp is supposed to have THC content below 0.2\u20131%, which depends on the legislation of different countries, while marijuana could reach THC content up to 20 to 30% in dry <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Inflorescence\" title=\"Inflorescence\" class=\"wiki-link\" data-key=\"d8122354225ee6a932a086ca8a1b999f\">inflorescences<\/a>.<sup id=\"rdp-ebb-cite_ref-SchillingCanna20_14-0\" class=\"reference\"><a href=\"#cite_note-SchillingCanna20-14\">[14]<\/a><\/sup> In 2015, Sawler <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SawlerTheGen15_15-0\" class=\"reference\"><a href=\"#cite_note-SawlerTheGen15-15\">[15]<\/a><\/sup> determined that hemp and marijuana significantly differ at the genome level, that different marijuana types are often not genetically close, and that THC is not related to the genetic distinction between hemp and marijuana. Hemp has been used for food and fibers, while marijuana was mostly used in traditional medicine.<sup id=\"rdp-ebb-cite_ref-Janatov.C3.A1Yield18_16-0\" class=\"reference\"><a href=\"#cite_note-Janatov.C3.A1Yield18-16\">[16]<\/a><\/sup> However, marijuana was <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Legality_of_cannabis\" title=\"Legality of cannabis\" class=\"wiki-link\" data-key=\"b80036537100734fce629cc441653451\">prohibited and criminalized<\/a> all around the world due to the psychoactive nature of THC. In 1988, the United Nations Convention against Illicit Traffic in Narcotic Drugs and Psychotropic Substances prohibited the use, production, and cultivation of <i>Cannabis<\/i> plants, which was recognized as a narcotic drug with psychotropic compounds, still posing a major problem in the legalization of cannabis with higher THC content.<sup id=\"rdp-ebb-cite_ref-AguilarMedicinal18_17-0\" class=\"reference\"><a href=\"#cite_note-AguilarMedicinal18-17\">[17]<\/a><\/sup> Due to many different research studies based on the positive effects of cannabis on numerous diseases, more and more countries are slowly revising their legislation in favor of growing <i>Cannabis<\/i> plants for medical and scientific purposes in restricted area.\n<\/p><p><i>Cannabis<\/i> varieties (strains) are often not fully inbred; therefore, they have a relatively high level of heterogeneity and instability, compared to other crops. Different phenotypes can be found within one variety. One of the major breeding challenges is that <i>Cannabis<\/i> plants are mostly <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Dioecy\" title=\"Dioecy\" class=\"wiki-link\" data-key=\"8b14a8df69095b55b0fa5e63539d676b\">dioecious<\/a>, and they cannot pollinate themselves; hence, outcrossing commonly occurs.<sup id=\"rdp-ebb-cite_ref-EuphyticaCurrent04_18-0\" class=\"reference\"><a href=\"#cite_note-EuphyticaCurrent04-18\">[18]<\/a><\/sup> This paper aims to clarify the distinction between three different hemp varieties and their 11 phenotypes according to an analysis of cannabinoids and terpenoids. Three varieties\u2014Carmagnola Selected (CS), Tiborszallasi (TS), and Finola Selection (FS)\u2014were chosen because of the expressed interest of our partners from the industry. All of them are registered on the E.U. variety list and are grown as out-growing varieties. Together, we found out that different phenotypes could be detected within them, and we supposed that they could express different chemical profiles as well, with a further different application in pharmacy. That said, the ultimate objective was to establish a connection between the chemical composition and morphological characteristics of hemp plants and to identify phenotypes with an interesting ratio between cannabinoids for further pharmaceutical applications.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Differentiation_of_phenotypes_according_to_visual_traits\">Differentiation of phenotypes according to visual traits<\/span><\/h3>\n<p>In total, 11 different phenotypes were selected according to visual traits observed within varieties that were compared with each other, and reference types<sup id=\"rdp-ebb-cite_ref-CREACereali_19-0\" class=\"reference\"><a href=\"#cite_note-CREACereali-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ERALearnNational_20-0\" class=\"reference\"><a href=\"#cite_note-ERALearnNational-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FinnishFood_21-0\" class=\"reference\"><a href=\"#cite_note-FinnishFood-21\">[21]<\/a><\/sup> (certified types from breeders) were added for each variety (Table 1). However, this is a first preliminary comparison between different phenotypes within mentioned varieties. Photos of all 11 phenotypes are presented in Figures S3\u2013S14 (see Supplementary materials). According to the Slovenian Ministry of Agriculture, Forestry, and Food in 2020, there was 6.67 ha of Carmagnola, 0.86 ha of Tiborszallasi, and 19.77 ha of Finola.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><b>Table 1.<\/b> The 11 different phenotypes (CI, CII, TI, TII, TIII, TIV, TV, FI, FII, FIII, FIV) that are defined by six different visual traits: size, color, leaf size, inflorescences, anthocyanin coloration of leaf petiole, and branching. For each variety, reference types are added. Note that size is the comparison between the height of plants within phenotypes in each variety. Plant color was described as light, medium, or dark green. Branching is scales, with * meaning little branched, up to **** meaning highly branched. - = No data available.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Phenotype\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Size\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Color\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Leaf size\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Inflorescences\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Anthocyanin coloration of leaf petiole\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Branching\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Remarks\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; text-align: left;\" colspan=\"8\"><b>Carmagnola Selected<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tall\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Light\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Large\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">***\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CII\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tall\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dark\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">****\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reference type\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tall\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dark\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; text-align: left;\" colspan=\"8\"><b>Tiborszallasi<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tall\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">****\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TII\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dark\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">***\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TIII\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dark\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TIV\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dark\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Large\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Big\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">***\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Compact flowers\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TV\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Strong anthocyanin coloration of the whole plant\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reference type\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tall\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dark\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">****\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; text-align: left;\" colspan=\"8\"><b>Finola Selection<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">FI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tall\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dark\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Big\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">****\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">FII\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Big\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">***\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">FIII\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Light\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">****\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">FIV\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dark\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Big\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">****\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reference type\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Small-medium\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">***\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Five plants with the same observed traits within one phenotype were labeled and separately sampled. Hemp is an open-pollinated plant and, therefore, also more prone to non-uniformity. Pollen can disperse a few kilometers in relation to the wind direction<sup id=\"rdp-ebb-cite_ref-SmallAPrelim03_22-0\" class=\"reference\"><a href=\"#cite_note-SmallAPrelim03-22\">[22]<\/a><\/sup>, which could be one of the reasons for higher heterogeneity in <i>Cannabis<\/i> plant varieties. Additionally, newly bred populations and marijuana populations are more uniform and can be easily grouped by their desirable traits, such as high THC\/CBD level, high limonene, or other terpenoid levels.<sup id=\"rdp-ebb-cite_ref-SawlerTheGen15_15-1\" class=\"reference\"><a href=\"#cite_note-SawlerTheGen15-15\">[15]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Chemical_analysis_of_the_essential_oil_of_hemp_.28Cannabis_sativa_L..29\">Chemical analysis of the essential oil of hemp (<i>Cannabis sativa<\/i> L.)<\/span><\/h3>\n<p>Briefly, the 11 phenotypes from CS, TS, and FS contained 0.09\u20133.38 mL of essential oil per 100 g of air-dried flower (1.34 mL\/100 g, on average). FS achieved the highest average content of essential oil (2.81 mL\/100 g air-dried flower), compared to CS (0.38 mL\/100 g air-dried flower) and TS (0.54 mL\/100 g air-dried flower). The greatest relative difference between phenotypes within varieties was between CS phenotypes (p < 0.0001) in relation to essential oil. CI achieved 0.23 mL\/100 g of air-dried flower, while CII achieved 0.53 mL\/100 g air dry flower of essential oil. When comparing monoecious and dioecious varieties, we noted that Bertoli <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BertoliFibre10_6-1\" class=\"reference\"><a href=\"#cite_note-BertoliFibre10-6\">[6]<\/a><\/sup> discovered a higher content of essential oil in dioecious plants. Nissen <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-NissenCharact10_23-0\" class=\"reference\"><a href=\"#cite_note-NissenCharact10-23\">[23]<\/a><\/sup> also reported about low content of essential oil in the CS variety. Significant differences have been recorded between different hemp varieties according to essential oil and cannabinoid content in previous studies as well.<sup id=\"rdp-ebb-cite_ref-NovakEssent01_24-0\" class=\"reference\"><a href=\"#cite_note-NovakEssent01-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NovakComp03_25-0\" class=\"reference\"><a href=\"#cite_note-NovakComp03-25\">[25]<\/a><\/sup>\n<\/p><p>As expected, the most abundant terpenoids were <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Myrcene\" title=\"Myrcene\" class=\"wiki-link\" data-key=\"41d45def1bca53b072962a823709a0ad\">myrcene<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Caryophyllene\" title=\"Caryophyllene\" class=\"wiki-link\" data-key=\"2b57176900e3cbad5be1ea724ba37cfd\">\u03b2-caryophyllene<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Pinene\" title=\"Pinene\" class=\"wiki-link\" data-key=\"6de2eb2cc03bfd4404a442d2f0e4076b\">\u03b1-pinene<\/a>, and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Humulene\" title=\"Humulene\" class=\"wiki-link\" data-key=\"9c05eef99c1446cac40b10d57a75ccff\">\u03b1-humulene<\/a> in all three varieties.<sup id=\"rdp-ebb-cite_ref-NamdarVaria18_3-2\" class=\"reference\"><a href=\"#cite_note-NamdarVaria18-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BertoliFibre10_6-2\" class=\"reference\"><a href=\"#cite_note-BertoliFibre10-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NissenCharact10_23-1\" class=\"reference\"><a href=\"#cite_note-NissenCharact10-23\">[23]<\/a><\/sup> The proportion of the 10 main components of essential oil were compared with analysis of variance (ANOVA) and presented in superscripts in Table 2 (as well as Table S1, Supplementary materials), in which all 29 analyzed components are presented. Notable points about these analyzed components include:\n<\/p>\n<ul><li> According to ANOVA, phenotypes CI and CII showed significant differences based on \u03b1-pinene, \u03b2-pinene, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=3-Carene\" title=\"3-Carene\" class=\"wiki-link\" data-key=\"8be01cc16e4cce273cbddafc06b81e21\">3-Carene<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Terpinene\" title=\"Terpinene\" class=\"wiki-link\" data-key=\"3ec3ee781b6e0b7466b1777e5482c813\">terpinolene<\/a>, \u03b2-caryophyllene, \u03b1-humulene, caryophyllene oxide, \u03b2-eudesmol, and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Phytol\" title=\"Phytol\" class=\"wiki-link\" data-key=\"a0d9eb3112f64f0ce1c4b036eff64166\">phytol<\/a>. The CII phenotype had significantly higher contents of all main monoterpenes (\u03b1-pinene, \u03b2-pinene, 3-carene, and terpinolene) than phenotype CI, while the contents of all main sesquiterpenes (\u03b2-caryophyllene, \u03b1-humulene, caryophyllene oxide, \u03b2-eudesmol) and phytol were significantly higher in the CI phenotype. <\/li>\n<li> When comparing phenotypes of TS, differences were shown in caryophyllene oxide. <\/li>\n<li> Terpinolene and \u03b1-terpinene were found at different concentrations in FS phenotypes. As is evident from Table 2, the phenotypes of FS had very high proportions of limonene (4.1\u20135.2%), in comparison to CS and TS phenotypes, which is of great value since FS phenotypes also reached the highest content of essential oil. The lowest proportion of limonene was in the TI phenotype (0.6%). <\/li>\n<li> Phenotypes of FS had the highest proportions of \u03b1-terpineol, \u03b2-eudesmol, and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Bisabolol\" title=\"Bisabolol\" class=\"wiki-link\" data-key=\"89705d0817b73e1c72693af6678ce1cb\">\u03b1-bisabolol<\/a> and very low proportions of caryophyllene, oxide, and undetachable proportion of phytol, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Nerolidol\" title=\"Nerolidol\" class=\"wiki-link\" data-key=\"7ef6af4c2e9d806372ecc031ce042896\">cis-nerolidol<\/a>, etc. <\/li>\n<li> The highest proportion of \u03b1-pinene was in all TS phenotypes, especially in TIV (11.9%), and in phenotype CII (11.9%). A distinctly lower proportion was identified in FIV (0.7%). <\/li>\n<li> The highest proportion of myrcene was also observed in the five TS phenotypes; TI had the highest proportion among the TS phenotypes (29.9%). <\/li>\n<li> The highest proportion of, \u03b2-pinene, 3-carene, \u03b3-terpinene, terpinolene, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Borneol\" title=\"Borneol\" class=\"wiki-link\" data-key=\"221c1d78d4af25d79fca17d502d9c664\">borneol<\/a>, and menthol occurred in CII among all phenotypes, while the highest proportion of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cedrene\" title=\"Cedrene\" class=\"wiki-link\" data-key=\"f0c6258bfedd9d61a0634b7a83e4e26a\">\u03b1-cedrene<\/a>, \u03b2-caryophyllene, \u03b1-humulene, cis-nerolidol, geranyl isobutyrate, caryophyllene oxide, \u03b2-eudesmol, and phytol occurred in CI (almost all components of sesquiterpenes). However, the CS variety also had the lowest amount of essential oil compared to other investigated varieties. <\/li>\n<li> FIII had the highest proportion of \u03b1-terpinene (0.6%) and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Fenchone\" title=\"Fenchone\" class=\"wiki-link\" data-key=\"230d8f062f68ac3f3070c715b280f8e8\">fenchone<\/a> (0.2%), and TI had the highest proportion of <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=P-Cymene\" title=\"P-Cymene\" class=\"wiki-link\" data-key=\"06005b4dd1e60b101c46f20f2055e2c0\">p-cymene<\/a> (0.1%). <\/li>\n<li> The highest proportion of borneol (0.2%) and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Geranyl_acetate\" title=\"Geranyl acetate\" class=\"wiki-link\" data-key=\"19deb8c14100be38348b017b4d0eb903\">geranyl acetate<\/a> (0.3%) was in FIV. FII contained the most \u03b1-terpineol (1.0%). <\/li>\n<li> The terpenoids p-cymene, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Camphor\" title=\"Camphor\" class=\"wiki-link\" data-key=\"f39a08f933fd795c66be9718518af75a\">camphor<\/a>, isoborneol, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Citronellol\" title=\"Citronellol\" class=\"wiki-link\" data-key=\"ea92401fad6df8681b440ace124e0a67\">\u03b2-citronellol<\/a>, and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Neryl_acetate\" title=\"Neryl acetate\" class=\"wiki-link\" data-key=\"e82a9cf161492841119a49d57e32269d\">neryl acetate<\/a> had contents lower than 0.1% or were undetectable.<\/li><\/ul>\n<p><br \/>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"23\"><b>Table 2.<\/b> Average essential oil (EO) content (mL\/100 g air-dried flower) of the main components in the inflorescence and average composition (%) of essential oil. Groups (a, b, c, and d) were determined by analysis of variance (ANOVA) from different phenotypes of Carmagnola Selected, Tiborszallasi, and Finola Selection. The same letters present similarities between phenotypes, while different letters represent differences between phenotypes according to components of essential oil. The mean \u00b1 standard deviation (SD) is reported.\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Phenotype\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">CI\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">CII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TI\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TIII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TIV\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TV\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">FI\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">FII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">FIII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">FIV\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Average EO content\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.53\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.58\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.52\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.64\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.22\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.56\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.75\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.82\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.41\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.59\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.38\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b1- Pinene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.5<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.6<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.3<br \/>b,c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.4<br \/>a,b,c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.3<br \/>a,b,c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.9<br \/>d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.5<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.7<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.0<br \/>a,b,c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.5<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.7<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b2-Pinene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.1<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.0<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.9<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.3<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.5<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.3<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.6<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.8<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.3<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.7\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Myrcene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.5<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20.9<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29.9<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">21.4<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25.2<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26.7<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24.9<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19.2<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">21.7<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.0<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.1<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.6\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Limonene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.6<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.3<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.8<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.2<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.1<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.4<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.2<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.7\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Terpinolene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.9<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13.4<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.7<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.8<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.6<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.8<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.2<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.5<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.6<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.0<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.9<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b2-Caryophyllene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">21.3<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.5<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10.8<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.1<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.3<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.9<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.1<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19.7<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.5<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13.6<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19.7<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.9\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b1-Humulene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.8<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.5<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.0<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.6<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.7<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.1<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.0<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.4<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.9<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.2<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.7<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.7\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Caryophyllene oxide\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.9<br \/>d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.1<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.6<br \/>d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.2<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.6<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.3<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.3<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.3<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.3<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b2-Eudesmol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.8<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.3<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.22\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.31<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.09<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.47\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.41<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.24\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u03b1-Bisabolol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.7<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.6<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.2<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.8<br \/>d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.03<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.67\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.37<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.90\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.31<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.26\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Terpenoids have a great impact on human health.<sup id=\"rdp-ebb-cite_ref-RussoCannabis17_26-0\" class=\"reference\"><a href=\"#cite_note-RussoCannabis17-26\">[26]<\/a><\/sup> Almost all listed components have anti-inflammatory effects, including myrcene, \u03b2-caryophyllene, caryophyllene oxide, humulene, \u03b1-pinene, linalool, limonene, terpinolene, \u03b3-terpinene, nerolidol, borneol, fenchone, and \u03b2-eudesmol. Some components have positive impacts on cancer and tumor treatments, such as \u03b2-caryophyllene, which also has a synergistic anticancer effect with humulene. Iso-caryophyllene, humulene, \u03b1-pinene, and linalool have antitumor activity.<sup id=\"rdp-ebb-cite_ref-NuutinenMedic18_27-0\" class=\"reference\"><a href=\"#cite_note-NuutinenMedic18-27\">[27]<\/a><\/sup> All phenotypes of CS and TS had high proportions of caryophyllene oxide, which has a positive effect on type 1 and type 2 diabetes, cardiovascular diseases, and hypertension.<sup id=\"rdp-ebb-cite_ref-NuutinenMedic18_27-1\" class=\"reference\"><a href=\"#cite_note-NuutinenMedic18-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MudgeTheTerr19_28-0\" class=\"reference\"><a href=\"#cite_note-MudgeTheTerr19-28\">[28]<\/a><\/sup> CII had the highest proportion of \u03b2-pinene, which has an important role in the regulation of diabetes, cancer, obesity, and other chronic diseases. Myrcene, terpinolene, linalool, and nerolidol have sedative effects. Humulene and limonene treat depression, and humulene has also been used in traditional medicine for treating insomnia, anxiety, delirium, and depression. With a significantly higher proportion of \u03b1-humulene, the CI phenotype could be used in further pharmaceutical research, especially if the total amount of essential oil could be increased in breeding procedures or affected by the environment. This phenotype also had a significantly higher proportion of \u03b2-eudesmol, which stimulates appetite.<sup id=\"rdp-ebb-cite_ref-NuutinenMedic18_27-2\" class=\"reference\"><a href=\"#cite_note-NuutinenMedic18-27\">[27]<\/a><\/sup>\n<\/p><p>The total amount of detected monoterpenes was higher than that of sesquiterpenes in all three varieties.><sup id=\"rdp-ebb-cite_ref-BertoliFibre10_6-3\" class=\"reference\"><a href=\"#cite_note-BertoliFibre10-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MudgeTheTerr19_28-1\" class=\"reference\"><a href=\"#cite_note-MudgeTheTerr19-28\">[28]<\/a><\/sup> Comparing the investigated varieties, the highest monoterpene content (44.89%) occurred in TS, while CS varieties had the highest amount of sesquiterpenes (28.67%). In the group of monoterpenes belong compounds \u03b1-pinene, camphene, \u03b2-pinene, myrcene, 3-carene, \u03b1-terpinene, p-cimene, limonene, \u03b3-terpinene, fenchone, terpinolene, linalool, camphor, isoborneol, borneol, menthol, \u03b1-terpineol, \u03b2-citronellol, neryl acetate, and geranyl acetate, while in the group of sesquiterpenes belong compounds \u03b1-cedrene, \u03b2-caryophyllene, \u03b1-humulene, cis-nerolidol, geranyl isobutyrate, caryophyllene oxide, \u03b2-eudesmol, \u03b1-bisabolol, and phytol.\n<\/p><p>A difference between varieties was observed when the average of essential oil components in individual phenotypes was compared (Figure 1). Principal component 1 (PC1) explained 39.44%, and PC2 explained 25.90% of the variance. Phenotypes of FS and TS were clustered together, while phenotypes of CS were separated. Considering the average amount of compounds in essential oil phenotypes within FS and TS, they were more uniform than CS phenotypes. (More detailed PCA plots with all analyzed variables are presented in Figure S1, Supplementary materials.)\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig1_Er%C5%BEen_Plants21_10-5.png\" class=\"image wiki-link\" data-key=\"1fe68b5ba47bfcfd524657e95cd7a9aa\"><img alt=\"Fig1 Er\u017een Plants21 10-5.png\" src=\"https:\/\/www.cannaqa.wiki\/images\/thumb\/7\/74\/Fig1_Er%C5%BEen_Plants21_10-5.png\/600px-Fig1_Er%C5%BEen_Plants21_10-5.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> Principal component analysis (PCA) plots for phenotype averages according to essential oil components made on the first two PC scores (PC1 explained 39.44%, and PC2 explained 25.90%), with a total variance of 65.34%.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>According to all analyzed components of essential oil, each individual variety of CS, TS, and FS was analyzed separately. Significant differences between both phenotypes of CS that were found by ANOVA can clearly also be observed in the PCA (Figure 2). TS and FS phenotypes were not distinguishable, which means that these two varieties were more uniform than CS.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig2_Er%C5%BEen_Plants21_10-5.png\" class=\"image wiki-link\" data-key=\"412aebc721e7ea4eb60239815acd9a73\"><img alt=\"Fig2 Er\u017een Plants21 10-5.png\" src=\"https:\/\/www.cannaqa.wiki\/images\/thumb\/9\/94\/Fig2_Er%C5%BEen_Plants21_10-5.png\/600px-Fig2_Er%C5%BEen_Plants21_10-5.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> Principal component analysis (PCA) plots for phenotypes of the Carmagnola Selected variety, according to components of essential oil made on the first two PC scores (PC1 explained 42.63%, and PC2 explained 16.16%), with a total variance of 58.79%.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Chemical_analysis_of_cannabinoids_in_hemp_.28Cannabis_sativa_L..29_phenotypes\">Chemical analysis of cannabinoids in hemp (<i>Cannabis sativa<\/i> L.) phenotypes<\/span><\/h3>\n<p>Regarding the analysis of cannabinoids, 10 different cannabinoids were identified in the 11 included phenotypes using <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=High-performance_liquid_chromatography\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"4c5d83ffa9785383c3eb0be7ea78dd2b\">high-performance liquid chromatography<\/a> (HPLC) (Table 3). There was a significant difference in CBDA and CBGA cannabinoids between the CI and CII phenotypes. CII was more related to TS phenotypes. There were also significant differences in cannabinoids CBD and CBC between FS phenotypes and all other phenotypes.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"23\"><b>Table 3.<\/b> Average cannabinoid content (%) of Carmagnola Selected, Tiborszallasi, and Finola Selection. Groups (a, b, c, and d) were formed by analysis of variance (ANOVA) using different hemp phenotypes. The same letters present similarities between phenotypes, while different letters present differences between phenotypes. Values are presented as the mean \u00b1 standard deviation (SD).\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">Cannabinoid\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">CI\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">CII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TI\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TIII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TIV\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">TV\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">FI\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">FII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">FIII\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th>\n<th style=\"background-color:#dddddd; padding-left:10px; padding-right:10px;\">FIV\n<\/th>\n<th style=\"background-color:#eeeeee; padding-left:10px; padding-right:10px;\">SD\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.60<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.78<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.61<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.64<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.19\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBDA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.70<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.53\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.78<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.94\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.22<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.11<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.56\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.65<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.06\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.78<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.93<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.37\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.48<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.35\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.41<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.35\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.36<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.59<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.24\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBG\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04<br \/>b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06<br \/>d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06<br \/>d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBGA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.62<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.44<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.25<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.19<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.22<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.18<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.00<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.37\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.63<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.91<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.11<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.26\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u0394<sup>9<\/sup>-THC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04<br \/>a,b,c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06<br \/>b,c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.11<br \/>d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08<br \/>c,d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u0394<sup>9<\/sup>-THCA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.91<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.81\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.75<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.71<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.72\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.14<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.53<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.54\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.18<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.39<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.50<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.46<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.47<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.51<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.00<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01<br \/>a,b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06<br \/>d\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05<br \/>c\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBCA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.20<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.06\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.27<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.34<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.24<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.25<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.34<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.30<br \/>a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.59<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.64<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.64<br \/>b\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.10\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>CBDA represented the highest proportion of cannabinoids in all included phenotypes, with the highest content in the four FS phenotypes (6.36\u20136.59%). Compared to the other phenotypes, the highest proportion of CBGA was in phenotype CI (1.62%), followed by all FS phenotypes. Phenotype TV and CI had the highest proportion of \u0394<sup>9<\/sup>-THCA, while the highest proportions of \u0394<sup>9<\/sup>-THC were in the phenotypes of FS (0.08\u20130.11%). The lowest proportion of \u0394<sup>9<\/sup>-THCA and \u0394<sup>9<\/sup>-THC were in phenotypes TII and TIV. The CBN values were less than 0.04%, and \u0394<sup>8<\/sup>-THC could not be detected in any sample. \n<\/p><p>In prior research, Aizpurua-Olaizola <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Aizpurua-OlaizolaEvo16_29-0\" class=\"reference\"><a href=\"#cite_note-Aizpurua-OlaizolaEvo16-29\">[29]<\/a><\/sup> determined the highest proportion of \u0394<sup>9<\/sup>-THCA, CBDA, and CBGA in clones of mother plants from chemotypes I, II, and III in an unknown variety. Additionally, Glivar <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-GlivarCanna20_30-0\" class=\"reference\"><a href=\"#cite_note-GlivarCanna20-30\">[30]<\/a><\/sup> analyzed the cannabinoid content in 15 different hemp varieties; when comparing proportions of CBDA, CBD, THC, and \u0394<sup>9<\/sup>-THCA in bracts of TS, their values were higher than those found in our samples, while other proportions were comparable with our data. In CS, there were higher proportions of all components, compared to the components in our CS variety, except for \u0394<sup>9<\/sup>-THCA, which was higher in our CS samples.<sup id=\"rdp-ebb-cite_ref-GlivarCanna20_30-1\" class=\"reference\"><a href=\"#cite_note-GlivarCanna20-30\">[30]<\/a><\/sup>\n<\/p><p>As mentioned before, Small and Beckstead<sup id=\"rdp-ebb-cite_ref-SmallCommon73_11-1\" class=\"reference\"><a href=\"#cite_note-SmallCommon73-11\">[11]<\/a><\/sup> and Fournier <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-FournierIdent87_12-1\" class=\"reference\"><a href=\"#cite_note-FournierIdent87-12\">[12]<\/a><\/sup> have already determined five major chemotypes based on the chemical profile of different hemp plants, whether it is marijuana or hemp type; however, our classification further divides varieties in different phenotypes. Nevertheless, based on almost the same ratio between CBD and \u0394<sup>9<\/sup>-THC and CBDA and \u0394<sup>9<\/sup>-THCA, phenotypes CI and TV belong to chemotype II, while all other phenotypes were classified as chemotype III due to a high CBDA and low \u0394<sup>9<\/sup>-THCA content. None of the phenotypes were chemotype I, which was expected since the investigation included hemp and not marijuana varieties.\n<\/p><p>PCA plots for the averages of included phenotypes in all three varieties were performed according to cannabinoid content. PC1 explained 67.57%, and PC2 explained 15.70% of the variance seen in cannabinoid content. (More detailed PCA plots with all analyzed variables are presented in Figure S2, Supplementary materials.) There was a considerable difference between the phenotypes of FS compared to CS and TS, but we could not completely differentiate between varieties CS and TS. Nevertheless, there were greater differences between TS and CS phenotypes than between phenotypes within FS. The results are presented in Figure 3. Greater differences were observed based on essential oil proportions than proportions of cannabinoid content.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig3_Er%C5%BEen_Plants21_10-5.png\" class=\"image wiki-link\" data-key=\"c05a9613a3e43fd4c71d29d3c0dfa64f\"><img alt=\"Fig3 Er\u017een Plants21 10-5.png\" src=\"https:\/\/www.cannaqa.wiki\/images\/thumb\/0\/0a\/Fig3_Er%C5%BEen_Plants21_10-5.png\/600px-Fig3_Er%C5%BEen_Plants21_10-5.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3<\/b> Principal component analysis (PCA) plot averages of analyzed cannabinoids for included phenotypes made on the first two PC scores (PC1 explained 67.57%, and PC2 explained 14.70%), with a total variance of 82.27%.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Each variety was analyzed separately to identify any differences between phenotypes within varieties. As with the essential oil components, differences in cannabinoids between CS phenotypes (CI and CII) were detected, while there were no differences between the five TS and four FS phenotypes. An additional PCA was performed for the CS variety, and clear differences between the two phenotypes were observed (Figure 4). PC1 explained 41.89%, and PC2 explained 22.97% of the variance in cannabinoid content.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig4_Er%C5%BEen_Plants21_10-5.png\" class=\"image wiki-link\" data-key=\"f06f7400eb5dcd8572e33eca4356a2b8\"><img alt=\"Fig4 Er\u017een Plants21 10-5.png\" src=\"https:\/\/www.cannaqa.wiki\/images\/thumb\/b\/bb\/Fig4_Er%C5%BEen_Plants21_10-5.png\/600px-Fig4_Er%C5%BEen_Plants21_10-5.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 4<\/b> Principal component analysis (PCA) plots of cannabinoid content for the Carmagnola Selected variety without \u0394<sup>8<\/sup>-THC, made on the first two PC scores (PC1 explained 41.89%, and PC2 explained 22.97%), with a total variance of 64.86%.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Aizpurua-Olaizola <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Aizpurua-OlaizolaEvo16_29-1\" class=\"reference\"><a href=\"#cite_note-Aizpurua-OlaizolaEvo16-29\">[29]<\/a><\/sup> discovered differences between chemotypes and correlations with certain terpenoids and cannabinoids in different chemotypes. In that research, clones of different chemotypes were used. They discovered a higher similarity between chemotypes II and III than chemotype I. In our research, there were no evident differences between these two chemotypes. Terpenoids such as <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Selinene\" title=\"Selinene\" class=\"wiki-link\" data-key=\"1bab67302ec8e225a9fbf27fd0ca5a89\">\u03b3-selinene<\/a>, \u03b2-selinene, \u03b1-gurgujene, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Elemene\" title=\"Elemene\" class=\"wiki-link\" data-key=\"55ae181f694a8b23c4e0eb279d1e2236\">\u03b3-elemene<\/a>, selina-3,7(11)-diene, and \u03b2-curcumene had a higher correlation with THCA and a negative correlation with CBDA in chemotype I, while terpenoids like \u03b2-eudesmol, \u03b3-eudesmol, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Guaiol\" title=\"Guaiol\" class=\"wiki-link\" data-key=\"b58aac13dbb38fc6f8197be732042ec3\">guaiol<\/a>, <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Bisabolene\" title=\"Bisabolene\" class=\"wiki-link\" data-key=\"c08996fbd5a6011d51e43e8be8c2f7be\">\u03b1-bisabolene<\/a>, \u03b1-bisabolol, and <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Eucalyptol\" title=\"Eucalyptol\" class=\"wiki-link\" data-key=\"66f35d2eec87fdcae2ce463e2ba37160\">eucalyptol<\/a> had a higher correlation with CBDA and negative correlation with THCA in chemotype III.<sup id=\"rdp-ebb-cite_ref-Aizpurua-OlaizolaEvo16_29-2\" class=\"reference\"><a href=\"#cite_note-Aizpurua-OlaizolaEvo16-29\">[29]<\/a><\/sup> However, Hillig<sup id=\"rdp-ebb-cite_ref-HilligAChemo04_31-0\" class=\"reference\"><a href=\"#cite_note-HilligAChemo04-31\">[31]<\/a><\/sup> reported that the components \u03b2-eudesmol, \u03b3-eudesmol, and guaiol were characteristic of <i><a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_indica\" title=\"Cannabis indica\" class=\"wiki-link\" data-key=\"7ea23da7d68f4d25012bac401d531b8b\">C. indica<\/a><\/i>, which has a higher level of \u0394<sup>9<\/sup>-THC. In our case, the CI phenotype had the highest relative level of \u03b2-eudesmol and a high level of \u0394<sup>9<\/sup>-THCA. When comparing phenotypes of each variety separately, only CS phenotypes showed significant differences in CBDA and CBGA, while phenotypes of TS and FS were not distinguishable, except FI and FIV, where separation from other phenotypes within the variety was evident in PCA analysis. This also indicated that TS and FS varieties were more uniform than CS, although CS is a variety selected from Carmagnola, which is the oldest hemp landrace grown in Italy and has been used as a breeding parent for many new varieties.<sup id=\"rdp-ebb-cite_ref-EuphyticaCurrent04_18-1\" class=\"reference\"><a href=\"#cite_note-EuphyticaCurrent04-18\">[18]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Chemical_analysis_of_both_cannabinoids_and_essential_oil_in_hemp_.28Cannabis_sativa_L..29_phenotypes\">Chemical analysis of both cannabinoids and essential oil in hemp (<i>Cannabis sativa<\/i> L.) phenotypes<\/span><\/h3>\n<p>Due to the positive effects of <i><a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Cannabis_sativa\" title=\"Cannabis sativa\" class=\"wiki-link\" data-key=\"e003358742012354d1ff6002bc5781de\">C. sativa<\/a><\/i> on human health, it is slowly gaining acceptance as a medicine. Due to the human endocannabinoid system, cannabinoids bind to different receptors through which they act on different diseases.<sup id=\"rdp-ebb-cite_ref-AndreCannabis16_7-2\" class=\"reference\"><a href=\"#cite_note-AndreCannabis16-7\">[7]<\/a><\/sup> Numerous research studies have been conducted on the medicinal use of the <i>Cannabis<\/i> plant. Ware <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-WareTheMed05_32-0\" class=\"reference\"><a href=\"#cite_note-WareTheMed05-32\">[32]<\/a><\/sup> described 30 symptoms and diseases for which cannabis is used and most commonly reported. The most studied are multiple sclerosis, neuropathy, chronic pain, depression, arthritis, migraine, allergy, spinal pain asthma, and weight loss. Very important clinical studies were conducted also on treatments of several cancers, Alzheimer\u2019s disease, diabetes, epilepsy, Huntington\u2019s diseases, Tourette\u2019s syndrome, and also for treating HIV\/AIDS.<sup id=\"rdp-ebb-cite_ref-Aizpurua-OlaizolaEvo16_29-3\" class=\"reference\"><a href=\"#cite_note-Aizpurua-OlaizolaEvo16-29\">[29]<\/a><\/sup> The benefits of using <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Hemp\" title=\"Hemp\" class=\"wiki-link\" data-key=\"c23e30b6cf1df54f1dc338492c9f9da2\">industrial hemp<\/a> as medicine are in the low content of THC (lower than 0.3\u20131%), having no value as a recreational drug. Hemp with high CBD or other cannabinoid content, together with low THC content, could be available for general use without medical prescription and, on the other hand, farmers could grow industrial hemp for medicinal purposes without any special restrictions due to THC legislation. Consumption of hemp seeds has also positive effects on human health, especially because of the content of omega-3 and omega-6 fatty acids. Seeds contain also proteins, carbohydrates, oil and fibers, and minerals.<sup id=\"rdp-ebb-cite_ref-CallawayHemp04_33-0\" class=\"reference\"><a href=\"#cite_note-CallawayHemp04-33\">[33]<\/a><\/sup>\n<\/p><p>According to our results, specific phenotypes can be used for further breeding purposes with specific cannabinoid and\/or terpenoid content, which could be used as supplementary ingredients or in pharmacy. Lewis <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-LewisPharm18_34-0\" class=\"reference\"><a href=\"#cite_note-LewisPharm18-34\">[34]<\/a><\/sup> reported different chemovars of <i>Cannabis<\/i> with desirable cannabinoid and terpenoid profiles. Comparing our results with their research, the CII and TIV phenotypes had a high proportion of \u03b1-pinene and a high proportion of CBDA, which could be of interest due to their ability to improve learning and memory and as a modulator of THC overdose. TIV also had the most favorable ratio between total THC:CBD and high CBD content and \u0394<sup>9<\/sup>-THC under 0.2%, which is the limit for varieties registered on the E.U. variety list. This phenotype could be very interesting in the pharmaceutical industry for the production of CBD. Due to its low THC content, it could also be grown outdoors by farmers. Phenotype TV had a high proportion of \u03b1-pinene, a relatively high proportion of myrcene, and a THCA and CBDA ratio around 1:2. Myrcene is responsible for the sedative effects of cannabis, and together with THC, it causes the \u201ccouch-lock\u201d effect. All FS phenotypes had the highest amount of essential oil and cannabinoids, with especially high CBD, limonene, \u03b2-caryophyllene, and \u03b1-humulene proportions, which relieves pain and inflammation and treats addiction.<sup id=\"rdp-ebb-cite_ref-LewisPharm18_34-1\" class=\"reference\"><a href=\"#cite_note-LewisPharm18-34\">[34]<\/a><\/sup> In the past, cannabis inflorescences have been used in traditional medicine for the treatment of various diseases, such as acute pain, wound healing, insomnia, cough, and mania, which are also the subject of modern medicine studies related to cannabinoids.<sup id=\"rdp-ebb-cite_ref-JinSecond20_35-0\" class=\"reference\"><a href=\"#cite_note-JinSecond20-35\">[35]<\/a><\/sup>\n<\/p><p>Figure 5 explains Pearson\u2019s correlation coefficient of all three varieties and their phenotypes based on the main terpenoids and cannabinoids. Only a correlation coefficient higher than 0.7 was considered a significant correlation. As seen in Figure 5, \u03b1-pinene and \u03b2-pinene had a strong positive correlation. The cannabinoid CBDA had a strong positive correlation with the sesquiterpene \u03b1-bisabolol and cannabinoids CBD, CBC, and CBCA. Additionally, CBG showed a strong positive correlation with CBD and CBC. CBD had a negative correlation with caryophyllene oxide. Conversely, there was a positive correlation between CBG and cannabinoids CBD, CBC, and CBCA. CBC had a strong positive correlation with \u03b1-bisabolol and CBCA.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=File:Fig5_Er%C5%BEen_Plants21_10-5.png\" class=\"image wiki-link\" data-key=\"9e2be043b6a93473325dfa6982e7f65a\"><img alt=\"Fig5 Er\u017een Plants21 10-5.png\" src=\"https:\/\/www.cannaqa.wiki\/images\/thumb\/5\/57\/Fig5_Er%C5%BEen_Plants21_10-5.png\/1000px-Fig5_Er%C5%BEen_Plants21_10-5.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 5<\/b> Pearson\u2019s correlation coefficient of all three varieties and all phenotypes based on main terpenoids and cannabinoids.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Our research was based on outdoor hemp cultivation. Industrial hemp is typically cultivated outside, where growing conditions are different from indoor cultivation. Therefore, the results of outdoor cultivation can vary from those of hemp cultivated indoors, where plants are exposed to more uniform conditions. In indoor cultivation, growing temperatures, day length, light, fertilization, irrigation, pest control, and harvest time are tightly determined to be optimal for growth.<sup id=\"rdp-ebb-cite_ref-ChandraCanna17_36-0\" class=\"reference\"><a href=\"#cite_note-ChandraCanna17-36\">[36]<\/a><\/sup> Plants cultivated indoors are more stable and suitable for reproduction than plants grown outdoors.<sup id=\"rdp-ebb-cite_ref-Janatov.C3.A1Yield18_16-1\" class=\"reference\"><a href=\"#cite_note-Janatov.C3.A1Yield18-16\">[16]<\/a><\/sup> However, our purpose was to identify the most suitable phenotypes that could be grown outdoors with appropriate technology and contain a high cannabinoid and terpenoid content.<sup id=\"rdp-ebb-cite_ref-ChandraCanna17_36-1\" class=\"reference\"><a href=\"#cite_note-ChandraCanna17-36\">[36]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Materials_and_methods\">Materials and methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Plant_material\">Plant material<\/span><\/h3>\n<p>Three different dioecious hemp (<i>Cannabis sativa<\/i> L.) varieties (Cannabaceae) were grown in 2019: Carmagnola Selected (CS), Tiborszallasi (TS), and Finola Selection (FS). Two different phenotypes of CS (CI, CII), five phenotypes of TS (TI, TII, TIII, TIV, TV), and four phenotypes of FS (FI, FII, FIII, FIV) were selected according to visual traits (habitus, leaf size, length and compactness of the inflorescence, color of leaf petioles, and internode length) (Table 1). Plants were observed one to two times weekly throughout the whole growing season, and final phenotypes were determined at the end of maturity. The inflorescences of 10 plants of each phenotype were separately sampled according to their aforementioned morphological characteristics at their optimal maturity time. For chemical analysis, the inflorescences of five plants per phenotype were used. Plants began flowering in the middle of August and were sampled and harvested at the end of September when most of the pistils turned brown. Male plants were removed from the crop before flowering to prevent female pollination and seed development. The lowest average temperatures were in the middle of May (11.4 \u00b0C) and the highest temperatures were in June (22.7 \u00b0C). Temperatures were higher than usual except in May when temperatures were lower than usual. From June to September, there was more precipitation than in 30 years on average. The soil was fertilized with organic fertilizer before sowing. No irrigation and no pest control were used. Weeds were controlled only mechanically twice at the beginning of growing.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Standards_and_reagents\">Standards and reagents<\/span><\/h3>\n<p>The following terpenoid standards were purchased from Supelco: \u03b1-pinene, camphene, \u03b2-pinene, myrcene, 3-carene, \u03b1-terpinene, p-cymene, limonene, \u03b3-terpinene, fenchone, terpinolene, linalool, camphor, isoborneol, borneol, menthol, \u03b1-terpineol, \u03b2-citronellol, neryl acetate, geranyl acetate, \u03b1-cedrene, \u03b2-caryophyllene, \u03b1-humulene, cis-nerolidol, geranyl isobutyrate, caryophyllene oxide, \u03b2-eudesmol, \u03b1-bisabolol, and phytol. Cannabinoid standards were purchased from Sigma Aldrich (1 mg\/mL): cannabidiolic acid (CBDA), cannabigerolic acid (CBGA), cannabigerol (CBG), cannabidiol (CBD), cannabinol (CBN), \u0394<sup>9<\/sup>-tetrahydrocannabinol (\u0394<sup>9<\/sup>-THC), \u0394<sup>8<\/sup>-tetrahydrocannabinol (\u0394<sup>8<\/sup>-THC), cannabichromene (CBC), \u0394<sup>9<\/sup>-tetrahydrocannabinolic acid (\u0394<sup>9<\/sup>-THCA), and cannabichromenic acid (CBCA). Acid standards were dissolved in acetonitrile, and nonacid standards were dissolved in methanol. Hexane, ammonium format, and acetonitrile were purchased from Sigma Aldrich, methanol was purchased from J. T. Baker, and formic acid was purchased from Scharlau. Chemical structures of essential oils and cannabinoids were referenced from various sources<sup id=\"rdp-ebb-cite_ref-MCE_37-0\" class=\"reference\"><a href=\"#cite_note-MCE-37\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChemSpid_38-0\" class=\"reference\"><a href=\"#cite_note-ChemSpid-38\">[38]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Chem.C3.A9o_39-0\" class=\"reference\"><a href=\"#cite_note-Chem.C3.A9o-39\">[39]<\/a><\/sup> and are presented in the Supplementary materials.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sample_preparation\">Sample preparation<\/span><\/h3>\n<p>Harvested inflorescences without stems and the larger leaves were dried in the drier at 30 \u00b0C immediately after sample collection. Inflorescences were ground (30\u201350 g) and stored in the refrigerator at \u221220 \u00b0C until analysis. For <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Gas_chromatography\" title=\"Gas chromatography\" class=\"wiki-link\" data-key=\"5094f035cb8bead5003deb9181e398b8\">gas chromatography<\/a> (GC) with <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Chromatography_detector\" title=\"Chromatography detector\" class=\"wiki-link\" data-key=\"7787a159a1c7c0a3b2cc28bbc3189587\">flame ionization detection<\/a> (GC-FID), dried samples (27\u201330 g) of the 11 different phenotypes were steam distilled (3 hours, 1 L distilled water) by a Clevenger apparatus. Each plant within each phenotype was analyzed separately with five different plants per phenotype. The collected oil was diluted with 1 mL of hexane and separated with gas chromatography analysis. For HPLC analysis, ground flowers (200 \u00b1 9 mg) of 11 different phenotypes were weighed in 50 mL centrifuge tubes, and 25.0 mL of 80% methanol was added before vortexing for 30 seconds. Extraction was performed by sonicating in a bath for 15 minutes and vortexed every five minutes.<sup id=\"rdp-ebb-cite_ref-MudgeLeaner17_40-0\" class=\"reference\"><a href=\"#cite_note-MudgeLeaner17-40\">[40]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"GC.E2.80.93FID_analysis\">GC\u2013FID analysis<\/span><\/h3>\n<p>An Agilent GC 6890 (Agilent) with an HP 1 capillary column (Crosslinkeed Methyl Silicone Gum) of 2.5 mm \u00d7 0.2 mm \u00d7 0.1 mm (Agilent, Santa Clara, CA, USA) was used. The carrier gas flow (Helium 5.0) was set to 0.5 mL\/min. One \u00b5L of the solution was injected into the injector port at a temperature of 180 \u00b0C. The oven temperature program was 3.5 minutes at 60 \u00b0C, 3.5 \u00b0C\/minute to 155 \u00b0C, and 30 \u00b0C\/minute to 300 \u00b0C. Detection was carried out on a flame ionization detector set at 300 \u00b0C with a 42.47 min runtime. Identification was made with ChemStation software for GC.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"HPLC_analysis\">HPLC analysis<\/span><\/h3>\n<p>For HPLC, an Agilent 1200 (Agilent, Santa Clara, CA, USA) was used. Separation was achieved on a Raptor ARC-18 (octadecylsilane) with a 2.7 \u00b5m, 150 mm \u00d7 4.6 mm ID (Restek, Bellefonte, PA, USA) column. Mobile phase A was composed of 5 mM ammonium formate and 0.1% formic acid in deionized water with a final pH of 3.1. Mobile phase B was composed of acetonitrile and 0.1% formic acid. 25% mobile phase A, and 75% mobile phase B. The flow rate was 1.5 mL\/minute, and an isocratic elution was in use. A calibration curve for each standard ranging from 5 to 250 \u00b5g\/mL was prepared with correlation coefficients higher than 0.9975. The extracts were filtered through a disposable syringe with a PTFE filter (0.45 \u00b5m). The injection volume was 5 \u00b5L. Detection took place at 228 nm with a 15 minute run time. Identification was carried out with ChemStation for LC.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Statistical_analyses\">Statistical analyses<\/span><\/h3>\n<p>Principal component analysis was performed using OriginPro 2021. Data were plotted based on the first two components. Multifactor ANOVA with Duncan test at a 95% confidential interval was performed in Statgraphics Centurion Software. Eleven phenotypes were compared according to the components of essential oil and cannabinoids, which were selected according to morphological differences between phenotypes. Means and standard deviations were calculated in Microsoft Excel.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>The purpose of our study was to compare the composition of essential oil and cannabinoids in 11 hemp phenotypes from three varieties based on morphological distinctness. To our knowledge, this is the first research to study differences in cannabinoid and essential oil content in various phenotypes within different varieties of hemp. Our study showed that based on the content of cannabinoids and essential oil, FS significantly and positively differed from CS and TS varieties, making it important for the hemp industry. Based on PCA analysis, there were no evident differences between phenotypes in FS and TS within each analyzed individual variety, while the two CS phenotypes were distinctive. By comparing the average content of the detected compounds in essential oil and cannabinoids between individual phenotypes of all three varieties, differences were determined. Among all analyzed phenotypes, CI, TII, TIV, and TV showed an interesting ratio between THC and CBD for further use in the pharmaceutical industry and breeding. CI and TV had a similar ratio of THC and CBD, while TII and TIV had a very high level of CBD, which may be useful in the pharmaceutical industry for epilepsy treatment.<sup id=\"rdp-ebb-cite_ref-NeubauerCanna18_41-0\" class=\"reference\"><a href=\"#cite_note-NeubauerCanna18-41\">[41]<\/a><\/sup> Additionally, the phenotypes of FS could be interesting due to a high CBD content, which could be extracted. When comparing phenotypes, according to ANOVA, significant differences in individual components of essential oil and cannabinoids were shown, potentiating the use of individual phenotypes in specialized breeding programs or disease treatments. Cannabinoids and terpenoids may have synergistic effects on different disease symptoms. When breeding or growing selected phenotypes, the plant varieties will be more homogenous regarding their chemical composition and, consequently, more interesting for pharmaceutical use. The cultivation of <i>C. sativa<\/i> with a low content of \u0394<sup>9<\/sup>-THC gives farmers an opportunity to cultivate hemp outdoor under controlled conditions for pharmaceutical purposes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_materials\">Supplementary materials<\/span><\/h2>\n<p>The following files are held within this <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.mdpi.com\/article\/10.3390\/plants10050966\/s1\">downloadable .zip file<\/a>:\n<\/p>\n<ul><li> Figure S1: Principal component analysis (PCA) plot for three different hemp populations (Carmagnola Selected, Tiborszallasi, and Finola Selection) according to 29 components of essential oil, made on first two PC scores (PC1 explained 24.43%, and PC2 explained 17.53%), with a total variance of 41.96%<\/li>\n<li> Figure S2: PCA plot for all three populations (Carmagnola Selected, Tiborszallasi, and Finola Selection) according to all cannabinoids made on first two PC scores (PC1 explained 50.71%, and PC2 explained 17.00%) with a total variance of 67.71%<\/li>\n<li> Figure S3: PCA plot for all analyzed samples of all populations according to components of essential oil and cannabinoids made on the first two PC scores (PC1 explained 27.38%, and PC2 explained 16.20%) with a total variance of 43.58%<\/li>\n<li> Figure S4: Phenotype CI: Plants of CI were light green with large primary leaves<\/li>\n<li> Figure S5: Phenotype CII: Plants of the CII phenotype were dark green with narrow leaflets and anthocyanin coloration on the leaf petiole. Most plants had more lateral branches<\/li>\n<li> Figure S6: Phenotype TI: TI plants were taller compared to the other phenotypes with lateral branches and smaller inflorescences<\/li>\n<li> Figure S7: Phenotype TII: TII plants were dark green with anthocyanin coloration of leaf petioles on the primary and secondary branches <\/li>\n<li> Figure S8: Phenotype TIII: TIII plants were also dark green, smaller than other phenotypes, with less lateral branches, anthocyanin coloration of the leaf petioles, and pink bracts<\/li>\n<li> Figure S9: Phenotype TIV: TIV plants were dark green with big compact inflorescences and thin leaflets around the inflorescences<\/li>\n<li> Figure S10: TV plants were small and had strong anthocyanin coloration of upper and lower leaves, inflorescences, and leaf petioles<\/li>\n<li> Figure S11: Phenotype FI: Plants of the FI phenotype were dark green with compact growth and taller than other phenotypes of FS. Plants had long side branches with longer and larger leaves and higher main inflorescences<\/li>\n<li> Figure S12: Phenotype FII: FII plants were medium green with shorter side branches and compact inflorescences surrounded by many leaflets<\/li>\n<li> Figure S13: Phenotype FIII: Plants from the FIII phenotype were light green with long side branches and a wide angle between them. Leaves had medium anthocyanin-colored edges, and inflorescences had green leaflets and were not compact<\/li>\n<li> Figure S14: Phenotype FIV: FIV plants were dark green with long inflorescences, and long, upright side branches. Leaf petioles had strong anthocyanin coloration<\/li>\n<li> Table S1: Average essential oil (EO) content (mL\/100 g of air-dried hemp) in the inflorescences, average composition (%) of essential oil, and groups (a, b, c, and d) from statistical analysis performed by analysis of variance (ANOVA) from different phenotypes of Carmagnola Selected, Tiborszallasi, and Finola Selection. The same letters present similarities between phenotypes, while different letters present differences between phenotypes according to components of essential oil. The mean \u00b1 standard deviation (SD) is reported<\/li>\n<li> Table S2: Chemical structures of components of essential oil and cannabinoids described in the research<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowdledgements\">Acknowdledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Conceptualization, A.\u010c., S.K. and M.E.; data curation, M.E.; formal analysis, M.E.; investigation, M.E.; methodology, I.J.K., M.O. and M.E.; resources, M.E. and A.\u010c.; supervision, A.\u010c. and I.J.K.; visualization, M.E. and A.\u010c.; writing\u2014original draft preparation, M.E.; writing\u2014review and editing, A.\u010c., I.J.K., M.O. and S.K. All authors have read and agreed to the published version of the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research was funded by the Slovenian Research Agency (ARRS).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HilligGenetic05-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HilligGenetic05_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hillig, K.W. (2005). \"Genetic evidence for speciation in <i>Cannabis<\/i> (Cannabaceae)\". <i>Genetic Resources and Crop Evolution<\/i> <b>52<\/b>: 161\u201380. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10722-003-4452-y\">10.1007\/s10722-003-4452-y<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genetic+evidence+for+speciation+in+%27%27Cannabis%27%27+%28Cannabaceae%29&rft.jtitle=Genetic+Resources+and+Crop+Evolution&rft.aulast=Hillig%2C+K.W.&rft.au=Hillig%2C+K.W.&rft.date=2005&rft.volume=52&rft.pages=161%E2%80%9380&rft_id=info:doi\/10.1007%2Fs10722-003-4452-y&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CriniAppli20-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CriniAppli20_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Crini, G.; Lichtfouse, E.; Chanet, G. et al. (2020). \"Applications of hemp in textiles, paper industry, insulation and building materials, horticulture, animal nutrition, food and beverages, nutraceuticals, cosmetics and hygiene, medicine, agrochemistry, energy production and environment: A review\". <i>Environmental Chemistry Letters<\/i> <b>18<\/b>: 1451\u201376. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10311-020-01029-2\">10.1007\/s10311-020-01029-2<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Applications+of+hemp+in+textiles%2C+paper+industry%2C+insulation+and+building+materials%2C+horticulture%2C+animal+nutrition%2C+food+and+beverages%2C+nutraceuticals%2C+cosmetics+and+hygiene%2C+medicine%2C+agrochemistry%2C+energy+production+and+environment%3A+A+review&rft.jtitle=Environmental+Chemistry+Letters&rft.aulast=Crini%2C+G.%3B+Lichtfouse%2C+E.%3B+Chanet%2C+G.+et+al.&rft.au=Crini%2C+G.%3B+Lichtfouse%2C+E.%3B+Chanet%2C+G.+et+al.&rft.date=2020&rft.volume=18&rft.pages=1451%E2%80%9376&rft_id=info:doi\/10.1007%2Fs10311-020-01029-2&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NamdarVaria18-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NamdarVaria18_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-NamdarVaria18_3-1\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-NamdarVaria18_3-2\">3.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Namdar, D.; Mazuz, M.; Ion, A. et al. (2018). \"Variation in the compositions of cannabinoid and terpenoids in Cannabis sativa derived from inflorescence position along the stem and extraction methods\". <i>Industrial Crops and Products<\/i> <b>113<\/b>: 376\u201382. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.indcrop.2018.01.060\">10.1016\/j.indcrop.2018.01.060<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Variation+in+the+compositions+of+cannabinoid+and+terpenoids+in+Cannabis+sativa+derived+from+inflorescence+position+along+the+stem+and+extraction+methods&rft.jtitle=Industrial+Crops+and+Products&rft.aulast=Namdar%2C+D.%3B+Mazuz%2C+M.%3B+Ion%2C+A.+et+al.&rft.au=Namdar%2C+D.%3B+Mazuz%2C+M.%3B+Ion%2C+A.+et+al.&rft.date=2018&rft.volume=113&rft.pages=376%E2%80%9382&rft_id=info:doi\/10.1016%2Fj.indcrop.2018.01.060&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TurnerQuant78-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TurnerQuant78_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Turner, J.C.; Hemphill, J.K.; Mahlberg, P.G. (1978). \"Quantitative Determination of Cannabinoids in Individual Glandular Trichomes of <i>Cannabis sativa<\/i> L. (Cannabaceae)\". <i>American Journal of Botany<\/i> <b>65<\/b> (10): 1103\u20136. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fj.1537-2197.1978.tb06177.x\">10.1002\/j.1537-2197.1978.tb06177.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quantitative+Determination+of+Cannabinoids+in+Individual+Glandular+Trichomes+of+%27%27Cannabis+sativa%27%27+L.+%28Cannabaceae%29&rft.jtitle=American+Journal+of+Botany&rft.aulast=Turner%2C+J.C.%3B+Hemphill%2C+J.K.%3B+Mahlberg%2C+P.G.&rft.au=Turner%2C+J.C.%3B+Hemphill%2C+J.K.%3B+Mahlberg%2C+P.G.&rft.date=1978&rft.volume=65&rft.issue=10&rft.pages=1103%E2%80%936&rft_id=info:doi\/10.1002%2Fj.1537-2197.1978.tb06177.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Hanu.C5.A1Phyto16-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Hanu.C5.A1Phyto16_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hanu\u0161, L.O.; Meyer, S.M.; Mu\u00f1oz, E. et al. (2016). \"Phytocannabinoids: A unified critical inventory\". <i>Natural Product Reports<\/i> <b>33<\/b>: 1357\u201392. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1039%2FC6NP00074F\">10.1039\/C6NP00074F<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Phytocannabinoids%3A+A+unified+critical+inventory&rft.jtitle=Natural+Product+Reports&rft.aulast=Hanu%C5%A1%2C+L.O.%3B+Meyer%2C+S.M.%3B+Mu%C3%B1oz%2C+E.+et+al.&rft.au=Hanu%C5%A1%2C+L.O.%3B+Meyer%2C+S.M.%3B+Mu%C3%B1oz%2C+E.+et+al.&rft.date=2016&rft.volume=33&rft.pages=1357%E2%80%9392&rft_id=info:doi\/10.1039%2FC6NP00074F&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BertoliFibre10-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BertoliFibre10_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-BertoliFibre10_6-1\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-BertoliFibre10_6-2\">6.2<\/a><\/sup> <sup><a href=\"#cite_ref-BertoliFibre10_6-3\">6.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bertoli, A.; Tozzi, S.; Pistelli, L. et al. (2010). \"Fibre hemp inflorescences: From crop-residues to essential oil production\". <i>Industrial Crops and Products<\/i> <b>32<\/b> (3): 329\u201337. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.indcrop.2010.05.012\">10.1016\/j.indcrop.2010.05.012<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fibre+hemp+inflorescences%3A+From+crop-residues+to+essential+oil+production&rft.jtitle=Industrial+Crops+and+Products&rft.aulast=Bertoli%2C+A.%3B+Tozzi%2C+S.%3B+Pistelli%2C+L.+et+al.&rft.au=Bertoli%2C+A.%3B+Tozzi%2C+S.%3B+Pistelli%2C+L.+et+al.&rft.date=2010&rft.volume=32&rft.issue=3&rft.pages=329%E2%80%9337&rft_id=info:doi\/10.1016%2Fj.indcrop.2010.05.012&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AndreCannabis16-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AndreCannabis16_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-AndreCannabis16_7-1\">7.1<\/a><\/sup> <sup><a href=\"#cite_ref-AndreCannabis16_7-2\">7.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Andre, C.M.; Hausman, J.-F.; Guerriero, G. (2016). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396\">\"<i>Cannabis sativa<\/i>: The plant of the thousand and one molecules\"<\/a>. <i>Frontiers in Plant Medicine<\/i> <b>7<\/b>: 19. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffpls.2016.00019\">10.3389\/fpls.2016.00019<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396\">PMC4740396<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26870049\">26870049<\/a><span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%27%27Cannabis+sativa%27%27%3A+The+plant+of+the+thousand+and+one+molecules&rft.jtitle=Frontiers+in+Plant+Medicine&rft.aulast=Andre%2C+C.M.%3B+Hausman%2C+J.-F.%3B+Guerriero%2C+G.&rft.au=Andre%2C+C.M.%3B+Hausman%2C+J.-F.%3B+Guerriero%2C+G.&rft.date=2016&rft.volume=7&rft.pages=19&rft_id=info:doi\/10.3389%2Ffpls.2016.00019&rft_id=info:pmc\/PMC4740396&rft_id=info:pmid\/26870049&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4740396&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RussoTaming11-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RussoTaming11_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Russo, E.B. (2011). \"Taming THC: potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\". <i>British Journal of Pharmacology<\/i> <b>163<\/b> (7): 1344\u201364. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1476-5381.2011.01238.x\">10.1111\/j.1476-5381.2011.01238.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Taming+THC%3A+potential+cannabis+synergy+and+phytocannabinoid-terpenoid+entourage+effects&rft.jtitle=British+Journal+of+Pharmacology&rft.aulast=Russo%2C+E.B.&rft.au=Russo%2C+E.B.&rft.date=2011&rft.volume=163&rft.issue=7&rft.pages=1344%E2%80%9364&rft_id=info:doi\/10.1111%2Fj.1476-5381.2011.01238.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DaSilvaBiological12-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DaSilvaBiological12_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">da Silva, A.C.R.; Lopes, P.M.; de Azevedo, M.M.B. et al. (2012). \"Biological Activities of a-Pinene and \u03b2-Pinene Enantiomers\". <i>Molecules<\/i> <b>17<\/b> (6): 6305-6316. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fmolecules17066305\">10.3390\/molecules17066305<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Biological+Activities+of+a-Pinene+and+%CE%B2-Pinene+Enantiomers&rft.jtitle=Molecules&rft.aulast=da+Silva%2C+A.C.R.%3B+Lopes%2C+P.M.%3B+de+Azevedo%2C+M.M.B.+et+al.&rft.au=da+Silva%2C+A.C.R.%3B+Lopes%2C+P.M.%3B+de+Azevedo%2C+M.M.B.+et+al.&rft.date=2012&rft.volume=17&rft.issue=6&rft.pages=6305-6316&rft_id=info:doi\/10.3390%2Fmolecules17066305&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AppendinoAntibact08-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AppendinoAntibact08_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Appendino, G.; Gibbons, S.; Giana, A. et al. (2008). \"Antibacterial Cannabinoids from <i>Cannabis sativa<\/i>: A Structure\u2212Activity Study\". <i>Journal of Natural Products<\/i> <b>71<\/b> (8): 1427\u201330. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fnp8002673\">10.1021\/np8002673<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Antibacterial+Cannabinoids+from+%27%27Cannabis+sativa%27%27%3A+A+Structure%E2%88%92Activity+Study&rft.jtitle=Journal+of+Natural+Products&rft.aulast=Appendino%2C+G.%3B+Gibbons%2C+S.%3B+Giana%2C+A.+et+al.&rft.au=Appendino%2C+G.%3B+Gibbons%2C+S.%3B+Giana%2C+A.+et+al.&rft.date=2008&rft.volume=71&rft.issue=8&rft.pages=1427%E2%80%9330&rft_id=info:doi\/10.1021%2Fnp8002673&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmallCommon73-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SmallCommon73_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-SmallCommon73_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Small, E.; Beckstead, H.D. (1973). \"Common cannabinoid phenotypes in 350 stocks of <i>Cannabis<\/i>\". <i>Lloydia<\/i> <b>36<\/b> (2): 144\u201365. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/4744553\">4744553<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Common+cannabinoid+phenotypes+in+350+stocks+of+%27%27Cannabis%27%27&rft.jtitle=Lloydia&rft.aulast=Small%2C+E.%3B+Beckstead%2C+H.D.&rft.au=Small%2C+E.%3B+Beckstead%2C+H.D.&rft.date=1973&rft.volume=36&rft.issue=2&rft.pages=144%E2%80%9365&rft_id=info:pmid\/4744553&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FournierIdent87-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FournierIdent87_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-FournierIdent87_12-1\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fournier, G.; Richez-Dumanois, C.; Duvezin, J. et al. (1987). \"Identification of a new chemotype in Cannabis sativa: cannabigerol-dominant plants, biogenetic and agronomic prospects\". <i>Planta Medica<\/i> <b>53<\/b> (3): 277\u201380. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1055%2Fs-2006-962705\">10.1055\/s-2006-962705<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/3628560\">3628560<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identification+of+a+new+chemotype+in+Cannabis+sativa%3A+cannabigerol-dominant+plants%2C+biogenetic+and+agronomic+prospects&rft.jtitle=Planta+Medica&rft.aulast=Fournier%2C+G.%3B+Richez-Dumanois%2C+C.%3B+Duvezin%2C+J.+et+al.&rft.au=Fournier%2C+G.%3B+Richez-Dumanois%2C+C.%3B+Duvezin%2C+J.+et+al.&rft.date=1987&rft.volume=53&rft.issue=3&rft.pages=277%E2%80%9380&rft_id=info:doi\/10.1055%2Fs-2006-962705&rft_id=info:pmid\/3628560&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PollioTheName16-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PollioTheName16_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pollio, A. (2016). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5531363\">\"The Name of Cannabis: A Short Guide for Nonbotanists\"<\/a>. <i>Cannabis and Cannabinoid Research<\/i> <b>1<\/b> (1): 234-238. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fcan.2016.0027\">10.1089\/can.2016.0027<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5531363\">PMC5531363<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28861494\">28861494<\/a><span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5531363\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5531363<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Name+of+Cannabis%3A+A+Short+Guide+for+Nonbotanists&rft.jtitle=Cannabis+and+Cannabinoid+Research&rft.aulast=Pollio%2C+A.&rft.au=Pollio%2C+A.&rft.date=2016&rft.volume=1&rft.issue=1&rft.pages=234-238&rft_id=info:doi\/10.1089%2Fcan.2016.0027&rft_id=info:pmc\/PMC5531363&rft_id=info:pmid\/28861494&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5531363&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchillingCanna20-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchillingCanna20_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schilling, S.; Melzer, R.; McCabe, P.F. (2020). \"<i>Cannabis sativa<\/i>\". <i>Current Biology<\/i> <b>30<\/b> (1): R8\u2013R9. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cub.2019.10.039\">10.1016\/j.cub.2019.10.039<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31910378\">31910378<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%27%27Cannabis+sativa%27%27&rft.jtitle=Current+Biology&rft.aulast=Schilling%2C+S.%3B+Melzer%2C+R.%3B+McCabe%2C+P.F.&rft.au=Schilling%2C+S.%3B+Melzer%2C+R.%3B+McCabe%2C+P.F.&rft.date=2020&rft.volume=30&rft.issue=1&rft.pages=R8%E2%80%93R9&rft_id=info:doi\/10.1016%2Fj.cub.2019.10.039&rft_id=info:pmid\/31910378&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SawlerTheGen15-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SawlerTheGen15_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-SawlerTheGen15_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sawler, J.; Stout, J.M.; Gardner, K.M. et al. (2015). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350\">\"The Genetic Structure of Marijuana and Hemp\"<\/a>. <i>PLoS One<\/i> <b>10<\/b> (8): e0133292. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0133292\">10.1371\/journal.pone.0133292<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350\">PMC4550350<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26308334\">26308334<\/a><span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Genetic+Structure+of+Marijuana+and+Hemp&rft.jtitle=PLoS+One&rft.aulast=Sawler%2C+J.%3B+Stout%2C+J.M.%3B+Gardner%2C+K.M.+et+al.&rft.au=Sawler%2C+J.%3B+Stout%2C+J.M.%3B+Gardner%2C+K.M.+et+al.&rft.date=2015&rft.volume=10&rft.issue=8&rft.pages=e0133292&rft_id=info:doi\/10.1371%2Fjournal.pone.0133292&rft_id=info:pmc\/PMC4550350&rft_id=info:pmid\/26308334&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4550350&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Janatov.C3.A1Yield18-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Janatov.C3.A1Yield18_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-Janatov.C3.A1Yield18_16-1\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Janatov\u00e1, A.; Fra\u0148kov\u00e1, A.; Tlusto\u0161, P. et al. (2018). \"Yield and cannabinoids contents in different cannabis (<i>Cannabis sativa<\/i> L.) genotypes for medical use\". <i>Industrial Crops and Products<\/i> <b>112<\/b>: 363\u201367. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.indcrop.2017.12.006\">10.1016\/j.indcrop.2017.12.006<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Yield+and+cannabinoids+contents+in+different+cannabis+%28%27%27Cannabis+sativa%27%27+L.%29+genotypes+for+medical+use&rft.jtitle=Industrial+Crops+and+Products&rft.aulast=Janatov%C3%A1%2C+A.%3B+Fra%C5%88kov%C3%A1%2C+A.%3B+Tlusto%C5%A1%2C+P.+et+al.&rft.au=Janatov%C3%A1%2C+A.%3B+Fra%C5%88kov%C3%A1%2C+A.%3B+Tlusto%C5%A1%2C+P.+et+al.&rft.date=2018&rft.volume=112&rft.pages=363%E2%80%9367&rft_id=info:doi\/10.1016%2Fj.indcrop.2017.12.006&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AguilarMedicinal18-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AguilarMedicinal18_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Aguilar, S.; Guti\u00e9rrez, V.; S\u00e1nchez, L. et al. (23 April 2018). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/idpc.net\/publications\/2018\/04\/medicinal-cannabis-policies-and-practices-around-the-world\">\"Medicinal cannabis policies and practices around the world\"<\/a>. International Drug Policy Consortium<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/idpc.net\/publications\/2018\/04\/medicinal-cannabis-policies-and-practices-around-the-world\">https:\/\/idpc.net\/publications\/2018\/04\/medicinal-cannabis-policies-and-practices-around-the-world<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 April 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Medicinal+cannabis+policies+and+practices+around+the+world&rft.atitle=&rft.aulast=Aguilar%2C+S.%3B+Guti%C3%A9rrez%2C+V.%3B+S%C3%A1nchez%2C+L.+et+al.&rft.au=Aguilar%2C+S.%3B+Guti%C3%A9rrez%2C+V.%3B+S%C3%A1nchez%2C+L.+et+al.&rft.date=23+April+2018&rft.pub=International+Drug+Policy+Consortium&rft_id=https%3A%2F%2Fidpc.net%2Fpublications%2F2018%2F04%2Fmedicinal-cannabis-policies-and-practices-around-the-world&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EuphyticaCurrent04-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-EuphyticaCurrent04_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-EuphyticaCurrent04_18-1\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ranalli, P. (2004). \"Current status and future scenarios of hemp breeding\". <i>Euphytica<\/i> <b>140<\/b>: 121\u201331. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10681-004-4760-0\">10.1007\/s10681-004-4760-0<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Current+status+and+future+scenarios+of+hemp+breeding&rft.jtitle=Euphytica&rft.aulast=Ranalli%2C+P.&rft.au=Ranalli%2C+P.&rft.date=2004&rft.volume=140&rft.pages=121%E2%80%9331&rft_id=info:doi\/10.1007%2Fs10681-004-4760-0&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CREACereali-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CREACereali_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.crea.gov.it\/web\/cerealicoltura-e-colture-industriali\">\"Cerealicoltura e Colture Industriali\"<\/a>. Consiglio per la ricerca in agricoltura e l'analisi dell'economia agraria<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.crea.gov.it\/web\/cerealicoltura-e-colture-industriali\">https:\/\/www.crea.gov.it\/web\/cerealicoltura-e-colture-industriali<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 April 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cerealicoltura+e+Colture+Industriali&rft.atitle=&rft.pub=Consiglio+per+la+ricerca+in+agricoltura+e+l%27analisi+dell%27economia+agraria&rft_id=https%3A%2F%2Fwww.crea.gov.it%2Fweb%2Fcerealicoltura-e-colture-industriali&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ERALearnNational-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ERALearnNational_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.era-learn.eu\/network-information\/organisations\/national-food-chain-safety-office-1\">\"National Food Chain Safety Office (NEBIH)\"<\/a>. <i>ERA-LEARN<\/i>. European Union<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.era-learn.eu\/network-information\/organisations\/national-food-chain-safety-office-1\">https:\/\/www.era-learn.eu\/network-information\/organisations\/national-food-chain-safety-office-1<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 April 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=National+Food+Chain+Safety+Office+%28NEBIH%29&rft.atitle=ERA-LEARN&rft.pub=European+Union&rft_id=https%3A%2F%2Fwww.era-learn.eu%2Fnetwork-information%2Forganisations%2Fnational-food-chain-safety-office-1&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FinnishFood-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FinnishFood_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ruokavirasto.fi\/en\/\">\"Finnish Food Authority\"<\/a>. Finnish Food Authority<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ruokavirasto.fi\/en\/\">https:\/\/www.ruokavirasto.fi\/en\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 April 2021<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Finnish+Food+Authority&rft.atitle=&rft.pub=Finnish+Food+Authority&rft_id=https%3A%2F%2Fwww.ruokavirasto.fi%2Fen%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmallAPrelim03-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmallAPrelim03_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Small, E.; Antle, T. (2003). \"A Preliminary Study of Pollen Dispersal in <i>Cannabis sativa<\/i> in Relation to Wind Direction\". <i>Journal of Industrial Hemp<\/i> <b>8<\/b> (2): 37-50. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1300%2FJ237v08n02_03\">10.1300\/J237v08n02_03<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Preliminary+Study+of+Pollen+Dispersal+in+%27%27Cannabis+sativa%27%27+in+Relation+to+Wind+Direction&rft.jtitle=Journal+of+Industrial+Hemp&rft.aulast=Small%2C+E.%3B+Antle%2C+T.&rft.au=Small%2C+E.%3B+Antle%2C+T.&rft.date=2003&rft.volume=8&rft.issue=2&rft.pages=37-50&rft_id=info:doi\/10.1300%2FJ237v08n02_03&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NissenCharact10-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NissenCharact10_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-NissenCharact10_23-1\">23.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nissen, L.; Zatta, A.; Stefanini, I. et al. (2010). \"Characterization and antimicrobial activity of essential oils of industrial hemp varieties (<i>Cannabis sativa<\/i> L.)\". <i>Fitoterapia<\/i> <b>81<\/b> (5): 413\u201319. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.fitote.2009.11.010\">10.1016\/j.fitote.2009.11.010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Characterization+and+antimicrobial+activity+of+essential+oils+of+industrial+hemp+varieties+%28%27%27Cannabis+sativa%27%27+L.%29&rft.jtitle=Fitoterapia&rft.aulast=Nissen%2C+L.%3B+Zatta%2C+A.%3B+Stefanini%2C+I.+et+al.&rft.au=Nissen%2C+L.%3B+Zatta%2C+A.%3B+Stefanini%2C+I.+et+al.&rft.date=2010&rft.volume=81&rft.issue=5&rft.pages=413%E2%80%9319&rft_id=info:doi\/10.1016%2Fj.fitote.2009.11.010&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NovakEssent01-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NovakEssent01_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Novak, J.; Zitteril-Eglseer, K.; Deans, S.G. et al. (2001). \"Essential oils of different cultivars of <i>Cannabis sativa<\/i> L. and their antimicrobial activity\". <i>Flavour and Fragrance Journal<\/i> <b>16<\/b> (4): 259-262. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fffj.993\">10.1002\/ffj.993<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Essential+oils+of+different+cultivars+of+%27%27Cannabis+sativa%27%27+L.+and+their+antimicrobial+activity&rft.jtitle=Flavour+and+Fragrance+Journal&rft.aulast=Novak%2C+J.%3B+Zitteril-Eglseer%2C+K.%3B+Deans%2C+S.G.+et+al.&rft.au=Novak%2C+J.%3B+Zitteril-Eglseer%2C+K.%3B+Deans%2C+S.G.+et+al.&rft.date=2001&rft.volume=16&rft.issue=4&rft.pages=259-262&rft_id=info:doi\/10.1002%2Fffj.993&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NovakComp03-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NovakComp03_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Novak, J.; Franz, C. (2003). \"Composition of the Essential Oils and Extracts of Two Populations of <i>Cannabis sativa<\/i> L. ssp. <i>spontanea<\/i> from Austria\". <i>Journal of Essential Oil Research<\/i> <b>15<\/b> (3): 158\u201360. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F10412905.2003.9712100\">10.1080\/10412905.2003.9712100<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Composition+of+the+Essential+Oils+and+Extracts+of+Two+Populations+of+%27%27Cannabis+sativa%27%27+L.+ssp.+%27%27spontanea%27%27+from+Austria&rft.jtitle=Journal+of+Essential+Oil+Research&rft.aulast=Novak%2C+J.%3B+Franz%2C+C.&rft.au=Novak%2C+J.%3B+Franz%2C+C.&rft.date=2003&rft.volume=15&rft.issue=3&rft.pages=158%E2%80%9360&rft_id=info:doi\/10.1080%2F10412905.2003.9712100&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RussoCannabis17-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RussoCannabis17_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Russo, E.B.; Marcu, J. (2017). \"Cannabis Pharmacology: The Usual Suspects and a Few Promising Leads\". <i>Advances in Pharmacology<\/i> <b>80<\/b>: 67\u2013134. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fbs.apha.2017.03.004\">10.1016\/bs.apha.2017.03.004<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28826544\">28826544<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis+Pharmacology%3A+The+Usual+Suspects+and+a+Few+Promising+Leads&rft.jtitle=Advances+in+Pharmacology&rft.aulast=Russo%2C+E.B.%3B+Marcu%2C+J.&rft.au=Russo%2C+E.B.%3B+Marcu%2C+J.&rft.date=2017&rft.volume=80&rft.pages=67%E2%80%93134&rft_id=info:doi\/10.1016%2Fbs.apha.2017.03.004&rft_id=info:pmid\/28826544&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NuutinenMedic18-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NuutinenMedic18_27-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-NuutinenMedic18_27-1\">27.1<\/a><\/sup> <sup><a href=\"#cite_ref-NuutinenMedic18_27-2\">27.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nuutinen, T. (2018). \"Medicinal properties of terpenes found in Cannabis sativa and Humulus lupulus\". <i>European Journal of Medicinal Chemistry<\/i> <b>157<\/b>: 198\u2013228. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ejmech.2018.07.076\">10.1016\/j.ejmech.2018.07.076<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30096653\">30096653<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medicinal+properties+of+terpenes+found+in+Cannabis+sativa+and+Humulus+lupulus&rft.jtitle=European+Journal+of+Medicinal+Chemistry&rft.aulast=Nuutinen%2C+T.&rft.au=Nuutinen%2C+T.&rft.date=2018&rft.volume=157&rft.pages=198%E2%80%93228&rft_id=info:doi\/10.1016%2Fj.ejmech.2018.07.076&rft_id=info:pmid\/30096653&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MudgeTheTerr19-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MudgeTheTerr19_28-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-MudgeTheTerr19_28-1\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mudge, E.M.; Brown, P.N.; Murch, S.J. (2019). \"The Terroir of Cannabis: Terpene Metabolomics as a Tool to Understand Cannabis sativa Selections\". <i>Planta Medica<\/i> <b>85<\/b> (9\u201310): 781\u201396. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1055%2Fa-0915-2550\">10.1055\/a-0915-2550<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/31096276\">31096276<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Terroir+of+Cannabis%3A+Terpene+Metabolomics+as+a+Tool+to+Understand+Cannabis+sativa+Selections&rft.jtitle=Planta+Medica&rft.aulast=Mudge%2C+E.M.%3B+Brown%2C+P.N.%3B+Murch%2C+S.J.&rft.au=Mudge%2C+E.M.%3B+Brown%2C+P.N.%3B+Murch%2C+S.J.&rft.date=2019&rft.volume=85&rft.issue=9%E2%80%9310&rft.pages=781%E2%80%9396&rft_id=info:doi\/10.1055%2Fa-0915-2550&rft_id=info:pmid\/31096276&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Aizpurua-OlaizolaEvo16-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Aizpurua-OlaizolaEvo16_29-0\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-Aizpurua-OlaizolaEvo16_29-1\">29.1<\/a><\/sup> <sup><a href=\"#cite_ref-Aizpurua-OlaizolaEvo16_29-2\">29.2<\/a><\/sup> <sup><a href=\"#cite_ref-Aizpurua-OlaizolaEvo16_29-3\">29.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aizpurua-Olaizola, O.; Soydaner, U.; \u00d6zt\u00fcrk, E. et al. (2016). \"Evolution of the Cannabinoid and Terpene Content during the Growth of Cannabis sativa Plants from Different Chemotypes\". <i>Journal of Natural Products<\/i> <b>79<\/b> (2): 324\u201331. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Facs.jnatprod.5b00949\">10.1021\/acs.jnatprod.5b00949<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26836472\">26836472<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evolution+of+the+Cannabinoid+and+Terpene+Content+during+the+Growth+of+Cannabis+sativa+Plants+from+Different+Chemotypes&rft.jtitle=Journal+of+Natural+Products&rft.aulast=Aizpurua-Olaizola%2C+O.%3B+Soydaner%2C+U.%3B+%C3%96zt%C3%BCrk%2C+E.+et+al.&rft.au=Aizpurua-Olaizola%2C+O.%3B+Soydaner%2C+U.%3B+%C3%96zt%C3%BCrk%2C+E.+et+al.&rft.date=2016&rft.volume=79&rft.issue=2&rft.pages=324%E2%80%9331&rft_id=info:doi\/10.1021%2Facs.jnatprod.5b00949&rft_id=info:pmid\/26836472&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GlivarCanna20-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GlivarCanna20_30-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-GlivarCanna20_30-1\">30.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Glivar, T.; Er\u017een, J.; Kreft, S. et al. (2020). \"Cannabinoid content in industrial hemp (<i>Cannabis sativa<\/i> L.) varieties grown in Slovenia\". <i>Industrial Crops and Products<\/i> <b>145<\/b>: 112082. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.indcrop.2019.112082\">10.1016\/j.indcrop.2019.112082<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabinoid+content+in+industrial+hemp+%28%27%27Cannabis+sativa%27%27+L.%29+varieties+grown+in+Slovenia&rft.jtitle=Industrial+Crops+and+Products&rft.aulast=Glivar%2C+T.%3B+Er%C5%BEen%2C+J.%3B+Kreft%2C+S.+et+al.&rft.au=Glivar%2C+T.%3B+Er%C5%BEen%2C+J.%3B+Kreft%2C+S.+et+al.&rft.date=2020&rft.volume=145&rft.pages=112082&rft_id=info:doi\/10.1016%2Fj.indcrop.2019.112082&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HilligAChemo04-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HilligAChemo04_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hillig, K.W. (2004). \"A chemotaxonomic analysis of terpenoid variation in <i>Cannabis<\/i>\". <i>Biochemical Systematics and Ecology<\/i> <b>32<\/b> (10): 875\u201391. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.bse.2004.04.004\">10.1016\/j.bse.2004.04.004<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+chemotaxonomic+analysis+of+terpenoid+variation+in+%27%27Cannabis%27%27&rft.jtitle=Biochemical+Systematics+and+Ecology&rft.aulast=Hillig%2C+K.W.&rft.au=Hillig%2C+K.W.&rft.date=2004&rft.volume=32&rft.issue=10&rft.pages=875%E2%80%9391&rft_id=info:doi\/10.1016%2Fj.bse.2004.04.004&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WareTheMed05-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WareTheMed05_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ware, M.A.; Adams, H.; Guy, G.W. (2005). \"The medicinal use of cannabis in the UK: Results of a nationwide survey\". <i>International Journal of Clinical Practice<\/i> <b>59<\/b> (3): 291\u20135. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1742-1241.2004.00271.x\">10.1111\/j.1742-1241.2004.00271.x<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15857325\">15857325<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+medicinal+use+of+cannabis+in+the+UK%3A+Results+of+a+nationwide+survey&rft.jtitle=International+Journal+of+Clinical+Practice&rft.aulast=Ware%2C+M.A.%3B+Adams%2C+H.%3B+Guy%2C+G.W.&rft.au=Ware%2C+M.A.%3B+Adams%2C+H.%3B+Guy%2C+G.W.&rft.date=2005&rft.volume=59&rft.issue=3&rft.pages=291%E2%80%935&rft_id=info:doi\/10.1111%2Fj.1742-1241.2004.00271.x&rft_id=info:pmid\/15857325&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CallawayHemp04-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CallawayHemp04_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Callaway, J.C. (2004). \"Hempseed as a nutritional resource: An overview\". <i>Euphytica<\/i> <b>140<\/b>: 65\u201372. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10681-004-4811-6\">10.1007\/s10681-004-4811-6<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hempseed+as+a+nutritional+resource%3A+An+overview&rft.jtitle=Euphytica&rft.aulast=Callaway%2C+J.C.&rft.au=Callaway%2C+J.C.&rft.date=2004&rft.volume=140&rft.pages=65%E2%80%9372&rft_id=info:doi\/10.1007%2Fs10681-004-4811-6&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LewisPharm18-34\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LewisPharm18_34-0\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-LewisPharm18_34-1\">34.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lewis, M.A.; Russo, E.B.; Smith, K.M. (2018). \"Pharmacological Foundations of Cannabis Chemovars\". <i>Planta Medica<\/i> <b>84<\/b> (4): 225\u201333. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1055%2Fs-0043-122240\">10.1055\/s-0043-122240<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29161743\">29161743<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pharmacological+Foundations+of+Cannabis+Chemovars&rft.jtitle=Planta+Medica&rft.aulast=Lewis%2C+M.A.%3B+Russo%2C+E.B.%3B+Smith%2C+K.M.&rft.au=Lewis%2C+M.A.%3B+Russo%2C+E.B.%3B+Smith%2C+K.M.&rft.date=2018&rft.volume=84&rft.issue=4&rft.pages=225%E2%80%9333&rft_id=info:doi\/10.1055%2Fs-0043-122240&rft_id=info:pmid\/29161743&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JinSecond20-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JinSecond20_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jin, D.; Dai, K.; Xie, Z. et al. (2020). <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7039888\">\"Secondary Metabolites Profiled in Cannabis Inflorescences, Leaves, Stem Barks, and Roots for Medicinal Purposes\"<\/a>. <i>Scientific Reports<\/i> <b>10<\/b> (1): 3309. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41598-020-60172-6\">10.1038\/s41598-020-60172-6<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7039888\">PMC7039888<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/32094454\">32094454<\/a><span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7039888\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC7039888<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Secondary+Metabolites+Profiled+in+Cannabis+Inflorescences%2C+Leaves%2C+Stem+Barks%2C+and+Roots+for+Medicinal+Purposes&rft.jtitle=Scientific+Reports&rft.aulast=Jin%2C+D.%3B+Dai%2C+K.%3B+Xie%2C+Z.+et+al.&rft.au=Jin%2C+D.%3B+Dai%2C+K.%3B+Xie%2C+Z.+et+al.&rft.date=2020&rft.volume=10&rft.issue=1&rft.pages=3309&rft_id=info:doi\/10.1038%2Fs41598-020-60172-6&rft_id=info:pmc\/PMC7039888&rft_id=info:pmid\/32094454&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC7039888&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChandraCanna17-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ChandraCanna17_36-0\">36.0<\/a><\/sup> <sup><a href=\"#cite_ref-ChandraCanna17_36-1\">36.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chandra, S.; Lata, H.; ElSohly, M.A. et al. (2017). \"Cannabis cultivation: Methodological issues for obtaining medical-grade product\". <i>Epilepsy & Behavior<\/i> <b>70<\/b> (Pt B): 302\u201312. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.yebeh.2016.11.029\">10.1016\/j.yebeh.2016.11.029<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28202406\">28202406<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis+cultivation%3A+Methodological+issues+for+obtaining+medical-grade+product&rft.jtitle=Epilepsy+%26+Behavior&rft.aulast=Chandra%2C+S.%3B+Lata%2C+H.%3B+ElSohly%2C+M.A.+et+al.&rft.au=Chandra%2C+S.%3B+Lata%2C+H.%3B+ElSohly%2C+M.A.+et+al.&rft.date=2017&rft.volume=70&rft.issue=Pt+B&rft.pages=302%E2%80%9312&rft_id=info:doi\/10.1016%2Fj.yebeh.2016.11.029&rft_id=info:pmid\/28202406&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MCE-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MCE_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.medchemexpress.com\/\">\"MedChemExpress\"<\/a>. MedChemExpress. 19 April 2021<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.medchemexpress.com\/\">https:\/\/www.medchemexpress.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MedChemExpress&rft.atitle=&rft.date=19+April+2021&rft.pub=MedChemExpress&rft_id=https%3A%2F%2Fwww.medchemexpress.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChemSpid-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChemSpid_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.chemspider.com\/\">\"ChemSpider\"<\/a>. Royal Society of Chemistry. 19 April 2021<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.chemspider.com\/\">https:\/\/www.chemspider.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ChemSpider&rft.atitle=&rft.date=19+April+2021&rft.pub=Royal+Society+of+Chemistry&rft_id=https%3A%2F%2Fwww.chemspider.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Chem.C3.A9o-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Chem.C3.A9o_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.chemeo.com\/\">\"Chem\u00e9o\"<\/a>. C\u00e9ondo GmbH. 19 April 2021<span class=\"printonly\">. <a target=\"_blank\" rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.chemeo.com\/\">https:\/\/www.chemeo.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chem%C3%A9o&rft.atitle=&rft.date=19+April+2021&rft.pub=C%C3%A9ondo+GmbH&rft_id=https%3A%2F%2Fwww.chemeo.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MudgeLeaner17-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MudgeLeaner17_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mudge, E.M.; Murch, S.J.; Brown, P.N. (2017). \"Leaner and greener analysis of cannabinoids\". <i>Analytical and Bioanalytical Chemistry<\/i> <b>409<\/b> (12): 3153\u201363. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00216-017-0256-3\">10.1007\/s00216-017-0256-3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Leaner+and+greener+analysis+of+cannabinoids&rft.jtitle=Analytical+and+Bioanalytical+Chemistry&rft.aulast=Mudge%2C+E.M.%3B+Murch%2C+S.J.%3B+Brown%2C+P.N.&rft.au=Mudge%2C+E.M.%3B+Murch%2C+S.J.%3B+Brown%2C+P.N.&rft.date=2017&rft.volume=409&rft.issue=12&rft.pages=3153%E2%80%9363&rft_id=info:doi\/10.1007%2Fs00216-017-0256-3&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NeubauerCanna18-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NeubauerCanna18_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Neubauer, D.; Benedik, M.P.; Osredkar, D. (2018). \"Cannabidiol for treatment of refractory childhood epilepsies: Experience from a single tertiary epilepsy center in Slovenia\". <i>Epilepsy & Behavior<\/i> <b>81<\/b>: 79\u201385. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.yebeh.2018.02.009\">10.1016\/j.yebeh.2018.02.009<\/a>. <a target=\"_blank\" rel=\"nofollow noreferrer noopener\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a target=\"_blank\" rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29526578\">29526578<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabidiol+for+treatment+of+refractory+childhood+epilepsies%3A+Experience+from+a+single+tertiary+epilepsy+center+in+Slovenia&rft.jtitle=Epilepsy+%26+Behavior&rft.aulast=Neubauer%2C+D.%3B+Benedik%2C+M.P.%3B+Osredkar%2C+D.&rft.au=Neubauer%2C+D.%3B+Benedik%2C+M.P.%3B+Osredkar%2C+D.&rft.date=2018&rft.volume=81&rft.pages=79%E2%80%9385&rft_id=info:doi\/10.1016%2Fj.yebeh.2018.02.009&rft_id=info:pmid\/29526578&rfr_id=info:sid\/en.wikipedia.org:Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20210705150551\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.751 seconds\nReal time usage: 0.803 seconds\nPreprocessor visited node count: 30918\/1000000\nPreprocessor generated node count: 34782\/1000000\nPost\u2010expand include size: 231482\/2097152 bytes\nTemplate argument size: 68697\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 644.344 1 - -total\n 87.55% 564.121 1 - Template:Reflist\n 74.09% 477.381 41 - Template:Citation\/core\n 67.15% 432.693 34 - Template:Cite_journal\n 10.56% 68.068 7 - Template:Cite_web\n 8.11% 52.282 52 - Template:Citation\/identifier\n 6.33% 40.803 1 - Template:Infobox_journal_article\n 5.94% 38.287 1 - Template:Infobox\n 4.02% 25.909 42 - Template:Citation\/make_link\n 3.95% 25.477 80 - Template:Infobox\/row\n-->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes\">https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t<\/body>","d78732feb44acdd119e6c7a504f8afa4_images":["https:\/\/www.cannaqa.wiki\/images\/thumb\/7\/74\/Fig1_Er%C5%BEen_Plants21_10-5.png\/1200px-Fig1_Er%C5%BEen_Plants21_10-5.png","https:\/\/www.cannaqa.wiki\/images\/thumb\/9\/94\/Fig2_Er%C5%BEen_Plants21_10-5.png\/1200px-Fig2_Er%C5%BEen_Plants21_10-5.png","https:\/\/www.cannaqa.wiki\/images\/thumb\/0\/0a\/Fig3_Er%C5%BEen_Plants21_10-5.png\/1200px-Fig3_Er%C5%BEen_Plants21_10-5.png","https:\/\/www.cannaqa.wiki\/images\/thumb\/b\/bb\/Fig4_Er%C5%BEen_Plants21_10-5.png\/1200px-Fig4_Er%C5%BEen_Plants21_10-5.png","https:\/\/www.cannaqa.wiki\/images\/thumb\/5\/57\/Fig5_Er%C5%BEen_Plants21_10-5.png\/2000px-Fig5_Er%C5%BEen_Plants21_10-5.png"],"d78732feb44acdd119e6c7a504f8afa4_timestamp":1625497550,"74856edf0e6bde68458770a07660ece5":{"type":"chapter","title":"1. Cannabis analyses","key":"74856edf0e6bde68458770a07660ece5"}},"link":"https:\/\/www.limswiki.org\/index.php\/Book:LIMSjournal_-_Summer_2021","price_currency":"","price_amount":"","book_size":"","download_url":"https:\/\/www.limsforum.com?ebb_action=book_download&book_id=97793","language":"","cta_button_content":"","toc":[{"type":"chapter","name":"1. Cannabis analyses","id":"74856edf0e6bde68458770a07660ece5","children":[{"type":"article","name":"Metabolomic analysis of cannabinoid and essential oil profiles in different hemp (Cannabis sativa L.) phenotypes (Er\u017een et al. 2021)","id":"d78732feb44acdd119e6c7a504f8afa4","pageUrl":"https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Metabolomic_analysis_of_cannabinoid_and_essential_oil_profiles_in_different_hemp_(Cannabis_sativa_L.)_phenotypes"},{"type":"article","name":"Accelerated solvent extraction of terpenes in cannabis coupled with various injection techniques for GC-MS analysis (Myers et al. 2021)","id":"a3b638adc3dd9824430235085d4b072b","pageUrl":"https:\/\/www.cannaqa.wiki\/index.php?title=Journal:Accelerated_solvent_extraction_of_terpenes_in_cannabis_coupled_with_various_injection_techniques_for_GC-MS_analysis"}]},{"type":"chapter","name":"2. Clinical and bioinformatics","id":"d434ecd441c8c22d9df227153518577c","children":[{"type":"article","name":"Diagnostic informatics: The role of digital health in diagnostic stewardship and the achievement of excellence, safety, and value (Georgiou et al. 2021)","id":"236fa37968f1db59b5299967fce1aa76","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Diagnostic_informatics:_The_role_of_digital_health_in_diagnostic_stewardship_and_the_achievement_of_excellence,_safety,_and_value"},{"type":"article","name":"Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results (Jin et al. 2021)","id":"cd03d5b8e3a613ce4ec49dfb1dc3ff59","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Development_and_implementation_of_an_LIS-based_validation_system_for_autoverification_toward_zero_defects_in_the_automated_reporting_of_laboratory_test_results"},{"type":"article","name":"Popularity and performance of bioinformatics software: The case of gene set analysis (Xie et al. 2021)","id":"7ef870605af2fae6ca37ea1b35850e50","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Popularity_and_performance_of_bioinformatics_software:_The_case_of_gene_set_analysis"}]},{"type":"chapter","name":"3. Data management and security","id":"a02b47d0feca425caeba4fad5d636358","children":[{"type":"article","name":"Blockchain-based healthcare workflow for IoT-connected laboratories in federated hospital clouds (Celesti et al. 2020)","id":"e49209f2666c8e463be219dc620f0ec5","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Blockchain-based_healthcare_workflow_for_IoT-connected_laboratories_in_federated_hospital_clouds"},{"type":"article","name":"Privacy-preserving healthcare informatics: A review (Chong 2021)","id":"5125b8e3cfcdb627ab689919e4e8a4b1","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Privacy-preserving_healthcare_informatics:_A_review"},{"type":"article","name":"Secure data outsourcing in presence of the inference problem: Issues and directions (Jebali et al. 2020)","id":"2537b4210c2770719a918d28c535508d","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Secure_data_outsourcing_in_presence_of_the_inference_problem:_Issues_and_directions"},{"type":"article","name":"Security and privacy in cloud-based eHealth systems (Sivan and Zukarnain 2021)","id":"89396bdb73ec18118ddadafe0171ee20","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Security_and_privacy_in_cloud-based_eHealth_systems"},{"type":"article","name":"Emerging and established trends to support secure health information exchange (Spanakis et al. 2021)","id":"edf532278ba56a75b0d284a94702804c","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Emerging_and_established_trends_to_support_secure_health_information_exchange"}]},{"type":"chapter","name":"4. Laboratory and scientific informatics","id":"2936237187d01cfa2a1c44845b2a3795","children":[{"type":"article","name":"Using interactive digital notebooks for bioscience and informatics education (Davies et al. 2020)","id":"31bea4f4ab2f82a24f0d0dad004bfbb0","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Using_interactive_digital_notebooks_for_bioscience_and_informatics_education"},{"type":"article","name":"The Application of Informatics to Scientific Work: Laboratory Informatics for Newbies (Liscouski 2021)","id":"d8b467af534a70312a21f63b61be26cd","pageUrl":"https:\/\/www.limswiki.org\/index.php\/LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies"},{"type":"article","name":"Informatics-driven quality improvement in the modern histology lab (Seifert et al. 2020)","id":"25073682c23be675cc4cd14be481b85b","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Informatics-driven_quality_improvement_in_the_modern_histology_lab"}]}],"settings":{"show_cover":"1","show_title":"1","show_subtitle":"0","show_full_title":"1","show_editor":"1","show_editor_pic":"1","show_publisher":"1","show_language":"1","show_size":"1","show_toc":"1","show_content_beneath_cover":"1","cta_button":"1","content_location":"1","toc_links":"disabled","log_in_msg":"<span><\/span> Please log in to read online.","cover_size":"medium"},"title_image":"https:\/\/s3.limsforum.com\/www.limsforum.com\/wp-content\/uploads\/Fig5_Jebali_JofInfoTelec2020_5-1.jpg"}}
LIMSjournal - Summer 2021
Volume 7, Issue 2
Editor: Shawn Douglas
Publisher: LabLynx Press
Copyright LabLynx Inc. All rights reserved.